2024-04-26T02:11:11.397 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T02:11:11.398 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} --name yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290 --verbose -- /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/orig.config.yaml 2024-04-26T02:11:11.432 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:11:11.515 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290 branch: wip-yuri2-testing-2024-04-24-0914-squid description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false job_id: '7674290' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri2-testing-2024-04-24-0914-squid ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 1 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false sha1: ddeaa91797339a19172c3036ff48cca58c12f448 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ddeaa91797339a19172c3036ff48cca58c12f448 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-04-24-0914-squid sha1: ddeaa91797339a19172c3036ff48cca58c12f448 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 1848 sha1: ddeaa91797339a19172c3036ff48cca58c12f448 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-04-24-0914-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ddeaa91797339a19172c3036ff48cca58c12f448 targets: smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOiklIl7i6IYpKW18rRudCtsCzg7lTdM95hlb6CkJd9VEXJlLIypWtuntlo94FgDoHJkLDbBhGiZp46ChoeGwbQ= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:23:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T02:11:11.515 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa; will attempt to use it 2024-04-26T02:11:11.516 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks 2024-04-26T02:11:11.516 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T02:11:11.517 INFO:teuthology.task.internal:Checking packages... 2024-04-26T02:11:11.541 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ddeaa91797339a19172c3036ff48cca58c12f448' 2024-04-26T02:11:11.541 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T02:11:11.541 INFO:teuthology.packaging:ref: None 2024-04-26T02:11:11.541 INFO:teuthology.packaging:tag: None 2024-04-26T02:11:11.541 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T02:11:11.541 INFO:teuthology.packaging:sha1: ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:11:11.541 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T02:11:11.687 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:11:11.688 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T02:11:11.698 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T02:11:11.698 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T02:11:11.711 INFO:teuthology.task.internal:Saving configuration 2024-04-26T02:11:11.723 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T02:11:11.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:11:11.756 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 02:05:39.722805', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIgAloThE29r3RGF/15sOuLD+2KxKqw+jQQeChtKyKX'} 2024-04-26T02:11:11.756 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T02:11:11.767 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-04-26T02:11:11.768 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T02:11:11.810 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa180f6ff70>, signals=[15]) 2024-04-26T02:11:11.811 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T02:11:11.817 INFO:teuthology.task.internal:Opening connections... 2024-04-26T02:11:11.817 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:11:11.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:11:11.897 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T02:11:11.905 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-26T02:11:12.026 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-26T02:11:12.027 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-26T02:11:12.074 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="Ubuntu" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="22.04" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_CODENAME=jammy 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:ID=ubuntu 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE=debian 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T02:11:12.075 INFO:teuthology.orchestra.run.smithi177.stdout:UBUNTU_CODENAME=jammy 2024-04-26T02:11:12.076 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-26T02:11:12.249 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T02:11:12.259 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T02:11:12.292 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T02:11:12.293 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-26T02:11:12.297 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T02:11:12.373 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T02:11:12.374 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-26T02:11:12.381 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T02:11:12.411 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T02:11:12.426 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T02:11:12.426 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T02:11:12.426 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T02:11:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:11:12.431 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-04-26T02:11:12.479 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-56-generic 2024-04-26T02:11:12.479 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.15.0-56-generic 2024-04-26T02:11:12.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-04-26T02:11:12.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-04-26T02:11:12.869 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:11:12.869 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T02:11:12.936 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T02:11:13.338 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T02:11:13.421 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T02:11:13.661 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T02:11:14.381 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T02:11:14.803 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T02:11:17.919 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T02:11:18.105 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T02:11:18.504 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T02:11:18.525 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T02:11:18.571 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T02:11:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T02:11:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T02:11:18.641 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T02:11:21.230 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T02:11:21.789 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:11:21.957 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T02:11:22.542 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T02:11:22.655 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T02:11:22.676 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T02:11:22.678 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T02:11:22.678 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T02:11:22.681 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T02:11:22.682 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T02:11:22.682 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T02:11:22.687 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T02:11:22.691 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T02:11:22.860 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T02:11:22.943 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T02:11:23.052 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T02:11:23.088 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T02:11:23.136 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T02:11:23.274 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T02:11:24.008 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:11:24.018 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T02:11:25.507 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T02:11:27.770 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T02:11:28.192 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T02:11:28.200 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T02:11:28.249 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T02:11:28.252 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T02:11:28.255 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T02:11:32.513 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 13.2 MB in 16s (850 kB/s) 2024-04-26T02:11:33.835 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:11:33.852 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:11:33.903 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:11:34.081 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-04-26T02:11:34.081 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-04-26T02:11:34.238 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:11:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:11:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:11:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:11:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T02:11:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: python2.7-minimal 2024-04-26T02:11:34.239 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:11:34.239 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-04-26T02:11:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:11:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T02:11:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T02:11:34.241 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-04-26T02:11:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T02:11:34.241 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-04-26T02:11:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: thermald 2024-04-26T02:11:34.281 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-04-26T02:11:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:11:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T02:11:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T02:11:34.282 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages will be upgraded: 2024-04-26T02:11:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T02:11:34.337 INFO:teuthology.orchestra.run.smithi177.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:11:34.338 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 113 MB of archives. 2024-04-26T02:11:34.338 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T02:11:34.338 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T02:11:34.931 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T02:11:35.134 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T02:11:36.366 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T02:11:36.367 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T02:11:36.367 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T02:11:36.570 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T02:11:36.614 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T02:11:37.003 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 113 MB in 2s (48.7 MB/s) 2024-04-26T02:11:37.172 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T02:11:39.193 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T02:11:39.198 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:11:39.268 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:11:41.361 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T02:11:41.381 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:11:41.462 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:11:41.930 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T02:11:41.950 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:11:41.988 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:11:48.752 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:11:48.854 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:11:49.170 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:11:49.291 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:11:49.611 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T02:11:49.622 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T02:11:49.660 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:11:55.714 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T02:11:55.737 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:11:55.781 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:11:58.350 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:11:58.484 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:11:58.835 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:11:59.100 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:11:59.207 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T02:11:59.336 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:12:00.650 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:12:00.650 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T02:12:00.650 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:12:00.650 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T02:12:00.821 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:12:02.274 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:12:03.657 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T02:12:03.783 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T02:12:03.909 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:12:03.978 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T02:12:03.978 INFO:teuthology.orchestra.run.smithi177.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:12:18.759 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T02:12:18.760 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub' 2024-04-26T02:12:18.777 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:12:18.795 INFO:teuthology.orchestra.run.smithi177.stdout:Generating grub configuration file ... 2024-04-26T02:12:19.104 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:12:19.125 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:12:19.407 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:12:19.413 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:12:19.493 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:12:19.499 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:12:19.686 INFO:teuthology.orchestra.run.smithi177.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:12:19.686 INFO:teuthology.orchestra.run.smithi177.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:12:19.686 INFO:teuthology.orchestra.run.smithi177.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:12:19.724 INFO:teuthology.orchestra.run.smithi177.stdout:done 2024-04-26T02:12:20.178 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-04-26T02:12:20.201 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.105.102 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:12:20.202 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-04-26T02:12:20.202 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T02:12:20.203 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T02:12:20.203 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T02:12:20.203 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T02:12:20.203 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T02:12:20.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-04-26T02:12:20.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-04-26T02:12:20.503 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:12:20.618 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:12:20.701 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:12:20.781 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:12:21.903 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:12:21.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:12:21.973 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:12:22.150 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-04-26T02:12:22.150 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:12:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:12:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:12:22.304 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:12:22.342 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:12:22.344 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-04-26T02:12:22.400 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-04-26T02:12:22.400 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-04-26T02:12:22.400 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.105.102 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:12:22.401 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-04-26T02:12:22.402 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-26T02:12:22.450 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.2YekrR65no 2024-04-26T02:12:22.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2YekrR65no 2024-04-26T02:12:22.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.2YekrR65no 2024-04-26T02:12:22.620 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.2YekrR65no is 10KB 2024-04-26T02:12:22.634 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.2YekrR65no 2024-04-26T02:12:22.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T02:12:22.694 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:12:22.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T02:12:22.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T02:12:22.761 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T02:12:22.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo update-grub 2024-04-26T02:12:23.874 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub' 2024-04-26T02:12:23.875 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:12:23.876 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-04-26T02:12:24.048 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:12:24.054 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:12:24.284 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:12:24.289 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:12:24.332 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:12:24.338 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:12:24.491 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:12:24.491 INFO:teuthology.orchestra.run.smithi177.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:12:24.491 INFO:teuthology.orchestra.run.smithi177.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:12:24.515 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-04-26T02:12:24.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-04-26T02:12:54.551 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T02:12:54.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-26T02:12:54.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:13:54.615 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T02:14:03.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-26T02:14:03.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:14:04.200 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-26T02:14:05.073 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-26T02:14:05.073 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T02:14:05.074 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-04-26T02:14:05.121 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-105-generic 2024-04-26T02:14:05.121 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T02:14:05.122 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T02:14:05.122 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T02:14:06.122 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T02:14:06.122 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T02:14:06.324 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-04-26T02:14:06.347 DEBUG:teuthology.parallel:result is None 2024-04-26T02:14:06.347 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T02:14:06.360 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T02:14:06.360 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T02:14:06.367 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T02:14:06.392 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T02:14:06.403 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T02:14:06.403 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T02:14:06.437 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T02:14:06.447 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T02:14:06.448 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T02:14:06.496 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:14:06.506 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:14:06.507 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T02:14:06.518 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T02:14:06.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T02:14:06.561 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T02:14:06.572 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T02:14:06.572 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T02:14:06.609 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T02:14:06.657 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T02:14:06.705 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:14:06.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T02:14:06.838 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-26T02:14:06.979 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T02:14:06.987 INFO:teuthology.task.internal:Starting timer... 2024-04-26T02:14:06.987 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T02:14:07.012 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T02:14:07.024 DEBUG:teuthology.task.selinux:Excluding smithi177: OS 'ubuntu' does not support SELinux 2024-04-26T02:14:07.025 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T02:14:07.025 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T02:14:07.025 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T02:14:07.025 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T02:14:07.038 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T02:14:07.043 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T02:14:07.169 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T02:14:07.185 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T02:14:07.186 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T02:21:05.368 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-04-26T02:21:05.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-26T02:21:05.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:21:05.452 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-26T02:21:05.534 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-26T02:21:05.534 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T02:21:05.546 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T02:21:05.546 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T02:21:05.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:21:05.604 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T02:21:05.604 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Command line: ntpd -gq 2024-04-26T02:21:05.604 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: ---------------------------------------------------- 2024-04-26T02:21:05.604 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: corporation. Support and training for ntp-4 are 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: available at https://www.nwtime.org/support 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: ---------------------------------------------------- 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: proto: precision = 0.060 usec (-24) 2024-04-26T02:21:05.605 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: basedate set to 2022-02-04 2024-04-26T02:21:05.606 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: gps base set to 2022-02-06 (week 2196) 2024-04-26T02:21:05.606 INFO:teuthology.orchestra.run.smithi177.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:21:05.606 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:21:05.606 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:21:05.607 INFO:teuthology.orchestra.run.smithi177.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:21:05.607 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen normally on 3 enp3s0f1 172.21.15.177:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen normally on 4 lo [::1]:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feea:b04b%5]:123 2024-04-26T02:21:05.608 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:05 ntpd[17197]: Listening on routing socket on fd #22 for interface updates 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:07.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:07 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:08.604 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:08 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:08.604 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:08 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:09 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:11.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:11 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:13.605 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:13.605 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:13.605 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stdout:26 Apr 02:21:13 ntpd[17197]: ntpd: time slew -0.003236 s 2024-04-26T02:21:13.606 INFO:teuthology.orchestra.run.smithi177.stdout:ntpd: time slew -0.003236s 2024-04-26T02:21:13.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T02:21:13.607 INFO:teuthology.orchestra.run.smithi177.stderr:26 Apr 02:21:13 ntpd[17197]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:21:13.652 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:21:13.653 INFO:teuthology.run_tasks:Running task install... 2024-04-26T02:21:13.663 DEBUG:teuthology.task.install:project ceph 2024-04-26T02:21:13.663 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}} 2024-04-26T02:21:13.663 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'} 2024-04-26T02:21:13.663 INFO:teuthology.task.install:Using flavor: default 2024-04-26T02:21:13.670 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T02:21:13.670 INFO:teuthology.task.install:extra packages: [] 2024-04-26T02:21:13.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-key list | grep Ceph 2024-04-26T02:21:13.752 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T02:21:13.782 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T02:21:13.782 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T02:21:13.783 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T02:21:13.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:21:13.935 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default/ 2024-04-26T02:21:13.936 INFO:teuthology.task.install.deb:Package version is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:21:14.172 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:21:14.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T02:21:14.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-04-26T02:21:14.409 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T02:21:14.420 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:21:14.420 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:21:14.430 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T02:21:14.453 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T02:21:14.460 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:21:14.474 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T02:21:14.500 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-26T02:21:14.503 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:21:14.789 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 28.9 kB in 0s (69.0 kB/s) 2024-04-26T02:21:15.810 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:21:15.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2457-gddeaa917-1jammy cephadm=19.0.0-2457-gddeaa917-1jammy ceph-mds=19.0.0-2457-gddeaa917-1jammy ceph-mgr=19.0.0-2457-gddeaa917-1jammy ceph-common=19.0.0-2457-gddeaa917-1jammy ceph-fuse=19.0.0-2457-gddeaa917-1jammy ceph-test=19.0.0-2457-gddeaa917-1jammy ceph-volume=19.0.0-2457-gddeaa917-1jammy radosgw=19.0.0-2457-gddeaa917-1jammy python3-rados=19.0.0-2457-gddeaa917-1jammy python3-rgw=19.0.0-2457-gddeaa917-1jammy python3-cephfs=19.0.0-2457-gddeaa917-1jammy python3-rbd=19.0.0-2457-gddeaa917-1jammy libcephfs2=19.0.0-2457-gddeaa917-1jammy libcephfs-dev=19.0.0-2457-gddeaa917-1jammy librados2=19.0.0-2457-gddeaa917-1jammy librbd1=19.0.0-2457-gddeaa917-1jammy rbd-fuse=19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:21:15.877 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-04-26T02:21:16.061 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-04-26T02:21:16.061 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-04-26T02:21:16.238 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:21:16.239 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:21:16.240 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-04-26T02:21:16.240 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T02:21:16.240 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T02:21:16.240 INFO:teuthology.orchestra.run.smithi177.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T02:21:16.240 INFO:teuthology.orchestra.run.smithi177.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:21:16.242 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet zip 2024-04-26T02:21:16.243 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-04-26T02:21:16.244 INFO:teuthology.orchestra.run.smithi177.stdout: btrfs-tools 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T02:21:16.343 INFO:teuthology.orchestra.run.smithi177.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:21:16.345 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T02:21:16.346 INFO:teuthology.orchestra.run.smithi177.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T02:21:16.470 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T02:21:16.470 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 179 MB of archives. 2024-04-26T02:21:16.470 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T02:21:16.470 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T02:21:16.652 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2457-gddeaa917-1jammy [3,545 kB] 2024-04-26T02:21:16.673 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T02:21:16.679 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T02:21:16.737 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T02:21:16.840 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T02:21:16.844 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T02:21:16.859 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T02:21:16.863 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T02:21:16.864 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T02:21:16.865 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T02:21:16.874 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T02:21:16.877 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T02:21:16.879 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T02:21:16.881 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T02:21:16.897 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T02:21:16.899 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T02:21:16.902 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T02:21:16.904 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T02:21:16.904 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T02:21:16.918 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T02:21:16.919 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T02:21:16.919 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T02:21:16.920 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T02:21:16.920 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T02:21:16.939 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T02:21:16.939 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T02:21:16.939 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T02:21:16.943 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T02:21:16.943 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T02:21:16.959 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T02:21:16.960 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T02:21:16.961 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T02:21:16.961 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T02:21:16.962 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T02:21:16.979 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T02:21:16.981 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T02:21:16.981 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T02:21:16.988 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T02:21:16.988 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T02:21:17.000 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T02:21:17.001 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T02:21:17.003 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T02:21:17.005 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T02:21:17.006 INFO:teuthology.orchestra.run.smithi177.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T02:21:17.022 INFO:teuthology.orchestra.run.smithi177.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T02:21:17.023 INFO:teuthology.orchestra.run.smithi177.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T02:21:17.026 INFO:teuthology.orchestra.run.smithi177.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T02:21:17.027 INFO:teuthology.orchestra.run.smithi177.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T02:21:17.028 INFO:teuthology.orchestra.run.smithi177.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T02:21:17.043 INFO:teuthology.orchestra.run.smithi177.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T02:21:17.109 INFO:teuthology.orchestra.run.smithi177.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T02:21:17.110 INFO:teuthology.orchestra.run.smithi177.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T02:21:17.111 INFO:teuthology.orchestra.run.smithi177.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T02:21:17.111 INFO:teuthology.orchestra.run.smithi177.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T02:21:17.113 INFO:teuthology.orchestra.run.smithi177.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T02:21:17.141 INFO:teuthology.orchestra.run.smithi177.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T02:21:17.144 INFO:teuthology.orchestra.run.smithi177.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T02:21:17.144 INFO:teuthology.orchestra.run.smithi177.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T02:21:17.168 INFO:teuthology.orchestra.run.smithi177.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T02:21:17.169 INFO:teuthology.orchestra.run.smithi177.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T02:21:17.178 INFO:teuthology.orchestra.run.smithi177.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T02:21:17.179 INFO:teuthology.orchestra.run.smithi177.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T02:21:17.179 INFO:teuthology.orchestra.run.smithi177.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T02:21:17.180 INFO:teuthology.orchestra.run.smithi177.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T02:21:17.180 INFO:teuthology.orchestra.run.smithi177.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T02:21:17.181 INFO:teuthology.orchestra.run.smithi177.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T02:21:17.185 INFO:teuthology.orchestra.run.smithi177.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T02:21:17.199 INFO:teuthology.orchestra.run.smithi177.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T02:21:17.202 INFO:teuthology.orchestra.run.smithi177.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T02:21:17.202 INFO:teuthology.orchestra.run.smithi177.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T02:21:17.227 INFO:teuthology.orchestra.run.smithi177.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T02:21:17.231 INFO:teuthology.orchestra.run.smithi177.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T02:21:17.232 INFO:teuthology.orchestra.run.smithi177.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T02:21:17.233 INFO:teuthology.orchestra.run.smithi177.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T02:21:17.238 INFO:teuthology.orchestra.run.smithi177.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T02:21:17.239 INFO:teuthology.orchestra.run.smithi177.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T02:21:17.243 INFO:teuthology.orchestra.run.smithi177.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T02:21:17.250 INFO:teuthology.orchestra.run.smithi177.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T02:21:17.251 INFO:teuthology.orchestra.run.smithi177.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T02:21:17.251 INFO:teuthology.orchestra.run.smithi177.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T02:21:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T02:21:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T02:21:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T02:21:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T02:21:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T02:21:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T02:21:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T02:21:17.638 INFO:teuthology.orchestra.run.smithi177.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2457-gddeaa917-1jammy [3,218 kB] 2024-04-26T02:21:17.981 INFO:teuthology.orchestra.run.smithi177.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2457-gddeaa917-1jammy [711 kB] 2024-04-26T02:21:18.055 INFO:teuthology.orchestra.run.smithi177.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2457-gddeaa917-1jammy [356 kB] 2024-04-26T02:21:18.103 INFO:teuthology.orchestra.run.smithi177.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2457-gddeaa917-1jammy [32.8 kB] 2024-04-26T02:21:18.111 INFO:teuthology.orchestra.run.smithi177.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2457-gddeaa917-1jammy [180 kB] 2024-04-26T02:21:18.121 INFO:teuthology.orchestra.run.smithi177.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2457-gddeaa917-1jammy [65.8 kB] 2024-04-26T02:21:18.122 INFO:teuthology.orchestra.run.smithi177.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2457-gddeaa917-1jammy [330 kB] 2024-04-26T02:21:18.164 INFO:teuthology.orchestra.run.smithi177.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2457-gddeaa917-1jammy [6,775 kB] 2024-04-26T02:21:18.571 INFO:teuthology.orchestra.run.smithi177.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2457-gddeaa917-1jammy [112 kB] 2024-04-26T02:21:18.576 INFO:teuthology.orchestra.run.smithi177.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2457-gddeaa917-1jammy [466 kB] 2024-04-26T02:21:18.591 INFO:teuthology.orchestra.run.smithi177.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2457-gddeaa917-1jammy [25.6 MB] 2024-04-26T02:21:19.548 INFO:teuthology.orchestra.run.smithi177.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2457-gddeaa917-1jammy [5,084 kB] 2024-04-26T02:21:19.644 INFO:teuthology.orchestra.run.smithi177.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2457-gddeaa917-1jammy [244 kB] 2024-04-26T02:21:19.652 INFO:teuthology.orchestra.run.smithi177.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2457-gddeaa917-1jammy [124 kB] 2024-04-26T02:21:19.657 INFO:teuthology.orchestra.run.smithi177.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2457-gddeaa917-1jammy [1,527 kB] 2024-04-26T02:21:19.707 INFO:teuthology.orchestra.run.smithi177.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2457-gddeaa917-1jammy [6,245 kB] 2024-04-26T02:21:19.918 INFO:teuthology.orchestra.run.smithi177.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2457-gddeaa917-1jammy [22.7 MB] 2024-04-26T02:21:20.803 INFO:teuthology.orchestra.run.smithi177.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2457-gddeaa917-1jammy [14.1 kB] 2024-04-26T02:21:20.803 INFO:teuthology.orchestra.run.smithi177.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2457-gddeaa917-1jammy [869 kB] 2024-04-26T02:21:20.820 INFO:teuthology.orchestra.run.smithi177.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2457-gddeaa917-1jammy [2,438 kB] 2024-04-26T02:21:20.862 INFO:teuthology.orchestra.run.smithi177.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2457-gddeaa917-1jammy [791 kB] 2024-04-26T02:21:20.871 INFO:teuthology.orchestra.run.smithi177.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2457-gddeaa917-1jammy [147 kB] 2024-04-26T02:21:20.873 INFO:teuthology.orchestra.run.smithi177.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2457-gddeaa917-1jammy [2,293 kB] 2024-04-26T02:21:20.938 INFO:teuthology.orchestra.run.smithi177.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2457-gddeaa917-1jammy [8,625 kB] 2024-04-26T02:21:21.225 INFO:teuthology.orchestra.run.smithi177.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2457-gddeaa917-1jammy [14.2 kB] 2024-04-26T02:21:21.226 INFO:teuthology.orchestra.run.smithi177.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2457-gddeaa917-1jammy [50.7 MB] 2024-04-26T02:21:23.679 INFO:teuthology.orchestra.run.smithi177.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2457-gddeaa917-1jammy [121 kB] 2024-04-26T02:21:23.681 INFO:teuthology.orchestra.run.smithi177.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2457-gddeaa917-1jammy [39.0 kB] 2024-04-26T02:21:23.682 INFO:teuthology.orchestra.run.smithi177.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2457-gddeaa917-1jammy [13.4 MB] 2024-04-26T02:21:25.073 INFO:teuthology.orchestra.run.smithi177.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2457-gddeaa917-1jammy [91.2 kB] 2024-04-26T02:21:25.639 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 179 MB in 9s (20.6 MB/s) 2024-04-26T02:21:25.770 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T02:21:25.868 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T02:21:25.874 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T02:21:25.944 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:21:26.280 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T02:21:26.306 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T02:21:26.347 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:21:26.691 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T02:21:26.717 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T02:21:26.758 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:21:27.120 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T02:21:27.146 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:21:27.186 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:21:27.581 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T02:21:27.607 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:21:27.647 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:21:28.008 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T02:21:28.035 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:21:28.075 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:21:28.370 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T02:21:28.396 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T02:21:28.428 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:21:28.723 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librados2. 2024-04-26T02:21:28.749 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../007-librados2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:28.789 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:29.259 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librbd1. 2024-04-26T02:21:29.285 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../008-librbd1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:29.325 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:29.761 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T02:21:29.788 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:29.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:30.097 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rados. 2024-04-26T02:21:30.112 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:30.155 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:30.483 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T02:21:30.509 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:21:30.550 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:30.802 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T02:21:30.817 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:30.860 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:31.138 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T02:21:31.164 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:21:31.196 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:31.474 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T02:21:31.501 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T02:21:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:21:31.860 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T02:21:31.887 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T02:21:31.926 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T02:21:32.196 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T02:21:32.222 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:32.262 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:32.582 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T02:21:32.609 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:21:32.876 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T02:21:32.903 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:21:33.245 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T02:21:33.261 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T02:21:33.303 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua5.1. 2024-04-26T02:21:33.666 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T02:21:33.706 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-any. 2024-04-26T02:21:34.019 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T02:21:34.051 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T02:21:34.570 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package zip. 2024-04-26T02:21:34.586 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T02:21:34.628 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T02:21:34.931 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package unzip. 2024-04-26T02:21:34.958 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T02:21:34.998 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:21:35.402 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package luarocks. 2024-04-26T02:21:35.429 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T02:21:35.468 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:21:35.826 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librgw2. 2024-04-26T02:21:35.852 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../025-librgw2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:36.412 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T02:21:36.438 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:36.478 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:36.822 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T02:21:36.849 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T02:21:36.889 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:21:37.192 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T02:21:37.218 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:37.258 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:37.620 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-common. 2024-04-26T02:21:37.646 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:37.678 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:38.966 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-base. 2024-04-26T02:21:38.993 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:39.033 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:39.804 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T02:21:39.830 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T02:21:39.870 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:21:40.173 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T02:21:40.200 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T02:21:40.239 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:21:40.567 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T02:21:40.594 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T02:21:40.634 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:21:40.895 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T02:21:40.921 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T02:21:40.962 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:21:41.248 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T02:21:41.274 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T02:21:41.314 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:21:41.592 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T02:21:41.619 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T02:21:41.659 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T02:21:41.903 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-portend. 2024-04-26T02:21:41.930 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T02:21:41.969 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T02:21:42.272 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T02:21:42.297 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T02:21:42.338 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T02:21:42.609 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T02:21:42.634 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T02:21:42.666 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:21:43.012 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T02:21:43.038 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T02:21:43.070 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T02:21:43.356 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T02:21:43.382 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T02:21:43.423 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T02:21:43.725 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T02:21:43.752 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T02:21:43.792 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:21:44.095 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-mako. 2024-04-26T02:21:44.121 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T02:21:44.161 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:21:44.464 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T02:21:44.491 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T02:21:44.531 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:21:44.775 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T02:21:44.802 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T02:21:44.842 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:21:45.153 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webob. 2024-04-26T02:21:45.179 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T02:21:45.219 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:21:45.572 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T02:21:45.599 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T02:21:45.642 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:21:45.950 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T02:21:45.977 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T02:21:46.016 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:21:46.346 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-paste. 2024-04-26T02:21:46.373 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T02:21:46.413 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:21:46.749 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T02:21:46.765 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T02:21:46.799 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:21:47.035 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T02:21:47.050 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T02:21:47.085 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:21:47.422 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T02:21:47.448 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T02:21:47.488 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T02:21:47.799 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T02:21:47.825 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T02:21:47.866 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:21:48.212 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T02:21:48.238 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T02:21:48.278 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T02:21:48.606 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-idna. 2024-04-26T02:21:48.633 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T02:21:48.673 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T02:21:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T02:21:48.969 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T02:21:49.000 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:21:49.279 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests. 2024-04-26T02:21:49.305 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T02:21:49.336 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:21:50.607 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T02:21:50.633 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T02:21:50.673 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:21:50.934 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T02:21:50.950 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:21:50.984 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:51.356 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T02:21:51.382 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:51.422 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:51.708 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T02:21:51.735 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T02:21:51.766 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:21:52.019 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T02:21:52.046 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T02:21:52.086 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T02:21:52.422 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T02:21:52.449 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:52.489 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:52.884 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T02:21:52.910 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:52.950 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:53.553 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T02:21:53.580 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:53.620 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:54.656 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph. 2024-04-26T02:21:54.672 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../066-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:54.714 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:54.984 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T02:21:55.007 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:55.042 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:55.420 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T02:21:55.447 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:55.486 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:55.915 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package cephadm. 2024-04-26T02:21:55.941 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../069-cephadm_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:21:55.981 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:56.284 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T02:21:56.310 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T02:21:56.350 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T02:21:56.686 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T02:21:56.705 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:21:56.744 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:21:57.757 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-babel. 2024-04-26T02:21:57.773 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:21:57.807 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:21:58.102 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T02:21:58.128 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T02:21:58.160 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:21:58.480 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T02:21:58.507 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:21:58.538 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:21:58.841 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T02:21:58.868 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T02:21:58.899 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T02:21:59.177 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-routes. 2024-04-26T02:21:59.203 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T02:21:59.243 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:21:59.547 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T02:21:59.573 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:21:59.613 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:01.143 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T02:22:01.159 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T02:22:01.193 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:22:01.646 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T02:22:01.662 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T02:22:01.696 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:22:02.049 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T02:22:02.077 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T02:22:02.116 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:22:02.427 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T02:22:02.454 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T02:22:02.493 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:22:03.235 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T02:22:03.263 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:22:03.386 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:04.290 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T02:22:04.317 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T02:22:04.349 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T02:22:04.643 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T02:22:04.670 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T02:22:04.700 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:22:05.071 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T02:22:05.087 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T02:22:05.129 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:22:05.424 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T02:22:05.451 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T02:22:05.490 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T02:22:05.802 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T02:22:05.829 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T02:22:05.877 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T02:22:06.172 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T02:22:06.199 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T02:22:06.230 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:22:06.516 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T02:22:06.543 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T02:22:06.582 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T02:22:06.869 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T02:22:06.896 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T02:22:06.935 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:22:07.566 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T02:22:07.582 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:22:07.624 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:07.943 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T02:22:07.971 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T02:22:08.010 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:22:08.271 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T02:22:08.299 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:22:08.338 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:22:08.599 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package jq. 2024-04-26T02:22:08.614 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:22:08.657 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T02:22:08.926 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package socat. 2024-04-26T02:22:08.953 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T02:22:08.992 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:22:09.288 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T02:22:09.315 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T02:22:09.354 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:22:09.717 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-test. 2024-04-26T02:22:09.744 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:22:09.783 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:12.087 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T02:22:12.114 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:22:12.154 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:12.449 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T02:22:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:22:12.515 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:12.818 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T02:22:12.845 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T02:22:12.884 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:22:13.205 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T02:22:13.221 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T02:22:13.280 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:22:13.593 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T02:22:13.620 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T02:22:13.659 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:22:14.064 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package pkg-config. 2024-04-26T02:22:14.092 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T02:22:14.130 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:22:14.383 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T02:22:14.400 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T02:22:14.433 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:22:14.903 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T02:22:14.931 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T02:22:14.977 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T02:22:15.481 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T02:22:15.508 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T02:22:15.555 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T02:22:15.909 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T02:22:15.937 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T02:22:15.975 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:22:16.286 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T02:22:16.314 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T02:22:16.353 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T02:22:16.706 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-py. 2024-04-26T02:22:16.733 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T02:22:16.781 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T02:22:17.109 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T02:22:17.136 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T02:22:17.175 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:22:17.671 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T02:22:17.698 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T02:22:17.737 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:22:18.040 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-toml. 2024-04-26T02:22:18.068 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T02:22:18.106 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T02:22:18.409 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T02:22:18.437 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T02:22:18.475 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:22:18.754 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T02:22:18.782 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T02:22:18.820 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:22:19.064 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T02:22:19.081 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T02:22:19.123 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:22:19.802 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package radosgw. 2024-04-26T02:22:19.830 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../116-radosgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:22:19.859 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:20.646 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T02:22:20.670 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:22:20.712 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:21.017 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T02:22:21.451 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:22:21.578 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:22:21.833 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T02:22:22.166 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T02:22:22.578 INFO:teuthology.orchestra.run.smithi177.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T02:22:22.653 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:23.177 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user cephadm....done 2024-04-26T02:22:23.326 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:22:23.602 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:22:23.845 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:22:23.971 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:22:24.206 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T02:22:24.456 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:22:24.574 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T02:22:24.876 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:22:25.253 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T02:22:25.504 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T02:22:25.755 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:22:25.982 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:22:26.232 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:22:26.500 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:22:26.751 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:27.010 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:22:27.261 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:22:27.396 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:22:27.505 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:22:27.739 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:22:28.089 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T02:22:28.130 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T02:22:28.192 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:22:28.318 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T02:22:28.678 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:22:28.789 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T02:22:29.048 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:22:29.308 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:22:29.617 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T02:22:29.743 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:22:30.420 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:22:30.679 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:22:30.780 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:22:30.882 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:22:31.182 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T02:22:31.584 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T02:22:31.838 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:22:32.194 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:22:32.471 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:22:32.814 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:22:33.065 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:22:33.182 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:33.475 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T02:22:33.793 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:22:35.014 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:22:35.266 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:22:35.401 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T02:22:35.660 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:22:35.761 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:22:35.862 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:22:36.163 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:22:36.398 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T02:22:36.799 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:22:36.941 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:22:37.243 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:22:37.485 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T02:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T02:22:37.979 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:22:38.222 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T02:22:38.465 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:22:38.591 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T02:22:38.867 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:22:38.993 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:22:39.294 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:22:39.637 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:22:39.887 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:22:40.022 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:22:40.148 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:22:40.238 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T02:22:40.483 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:22:40.609 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:22:41.002 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:22:41.261 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T02:22:41.387 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T02:22:41.638 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:22:41.764 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:22:42.073 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:22:42.383 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:22:42.642 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T02:22:42.768 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T02:22:43.052 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T02:22:43.329 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:22:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:22:44.083 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T02:22:44.368 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:22:44.704 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:22:44.805 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:44.906 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:45.032 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:22:45.368 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:45.494 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:45.620 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T02:22:45.905 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:46.007 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:46.124 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:46.340 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:22:46.340 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:22:46.775 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:46.910 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:47.069 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:47.378 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:22:48.732 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:48.842 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:48.951 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:49.077 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:49.203 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:49.720 INFO:teuthology.orchestra.run.smithi177.stdout:Adding group ceph....done 2024-04-26T02:22:50.097 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user ceph....done 2024-04-26T02:22:50.265 INFO:teuthology.orchestra.run.smithi177.stdout:Setting system user ceph properties....done 2024-04-26T02:22:50.285 INFO:teuthology.orchestra.run.smithi177.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T02:22:50.422 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T02:22:50.726 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T02:22:51.220 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:51.338 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:51.942 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:22:51.942 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:22:52.494 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:52.862 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T02:22:53.337 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:53.553 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:22:53.554 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:22:54.006 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:54.203 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:22:54.203 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:22:54.632 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:54.882 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:22:54.883 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:22:55.317 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:55.443 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:55.585 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:55.793 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:22:55.793 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:22:56.253 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:56.396 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:56.505 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:56.640 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:22:56.984 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T02:22:57.101 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T02:22:57.254 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T02:22:58.479 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T02:22:59.124 INFO:teuthology.orchestra.run.smithi177.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T02:22:59.130 DEBUG:teuthology.parallel:result is None 2024-04-26T02:22:59.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:22:59.277 DEBUG:teuthology.orchestra.run.smithi177:> dpkg-query -W -f '${Version}' ceph 2024-04-26T02:22:59.295 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:22:59.295 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:22:59.296 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457-gddeaa917-1jammy is installed. 2024-04-26T02:22:59.297 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T02:22:59.298 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:22:59.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T02:22:59.348 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T02:22:59.348 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:22:59.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T02:22:59.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T02:22:59.455 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T02:22:59.455 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:22:59.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T02:22:59.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T02:22:59.563 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T02:22:59.564 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:22:59.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T02:22:59.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T02:22:59.672 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T02:22:59.775 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T02:22:59.775 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-04-26T02:22:59.788 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T02:22:59.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T02:22:59.844 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T02:22:59.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T02:22:59.911 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T02:22:59.911 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-04-26T02:22:59.911 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290', 'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7674290', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'mon_bind_msgr2': False, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 1848, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'targets': {'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOiklIl7i6IYpKW18rRudCtsCzg7lTdM95hlb6CkJd9VEXJlLIypWtuntlo94FgDoHJkLDbBhGiZp46ChoeGwbQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_01:23:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T02:22:59.912 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T02:22:59.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T02:23:00.005 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:00.005 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T02:23:00.057 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T02:23:00.057 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-26T02:23:00.108 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T02:23:00.108 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:23:00.108 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-04-26T02:23:00.109 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:23:00.109 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-26 02:17:48.516178070 +0000 2024-04-26T02:23:00.109 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-26 02:17:48.208180821 +0000 2024-04-26T02:23:00.109 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-26 02:17:48.208180821 +0000 2024-04-26T02:23:00.109 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-04-26T02:23:00.109 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T02:23:00.163 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-26T02:23:00.164 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-26T02:23:00.164 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00044834 s, 1.1 MB/s 2024-04-26T02:23:00.165 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T02:23:00.214 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-26 02:17:49.040173390 +0000 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-26 02:17:48.776175748 +0000 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-26 02:17:48.776175748 +0000 2024-04-26T02:23:00.261 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-04-26T02:23:00.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T02:23:00.318 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-26T02:23:00.319 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-26T02:23:00.319 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000428034 s, 1.2 MB/s 2024-04-26T02:23:00.319 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T02:23:00.366 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-26 02:17:49.300171067 +0000 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-26 02:17:49.296171102 +0000 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-26 02:17:49.296171102 +0000 2024-04-26T02:23:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-04-26T02:23:00.414 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T02:23:00.467 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-26T02:23:00.467 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-26T02:23:00.467 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000360846 s, 1.4 MB/s 2024-04-26T02:23:00.468 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T02:23:00.518 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-26 02:17:50.048164386 +0000 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-26 02:17:49.804166565 +0000 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-26 02:17:49.804166565 +0000 2024-04-26T02:23:00.565 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-04-26T02:23:00.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T02:23:00.619 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-26T02:23:00.620 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-26T02:23:00.620 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000386877 s, 1.3 MB/s 2024-04-26T02:23:00.621 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T02:23:00.670 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:23:00.670 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-26T02:23:00.671 INFO:tasks.ceph:Generating config... 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2024-04-26T02:23:00.672 INFO:tasks.ceph:[global] ms type = async 2024-04-26T02:23:00.672 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T02:23:00.673 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T02:23:00.673 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T02:23:00.674 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T02:23:00.675 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T02:23:00.675 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T02:23:00.744 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T02:23:00.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T02:23:00.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T02:23:00.887 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.177:6789]'), ('mon.b', '[v1:172.21.15.177:6790]'), ('mon.c', '[v1:172.21.15.177:6791]')] 2024-04-26T02:23:00.887 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.177:6789],[v1:172.21.15.177:6790],[v1:172.21.15.177:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 1, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-26T02:23:00.888 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:00.888 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T02:23:00.934 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.177:6789]' --addv b '[v1:172.21.15.177:6790]' --addv c '[v1:172.21.15.177:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: generated fsid 13088462-cff2-40f2-960d-bd23a2ee33c5 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:setting min_mon_release = quincy 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:epoch 0 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:fsid 13088462-cff2-40f2-960d-bd23a2ee33c5 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:last_changed 2024-04-26T02:23:01.021078+0000 2024-04-26T02:23:01.019 INFO:teuthology.orchestra.run.smithi177.stdout:created 2024-04-26T02:23:01.021078+0000 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:min_mon_release 17 (quincy) 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:election_strategy: 1 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:0: v1:172.21.15.177:6789/0 mon.a 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:1: v1:172.21.15.177:6790/0 mon.b 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:2: v1:172.21.15.177:6791/0 mon.c 2024-04-26T02:23:01.020 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T02:23:01.021 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T02:23:01.069 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 13088462-cff2-40f2-960d-bd23a2ee33c5... 2024-04-26T02:23:01.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T02:23:01.141 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-04-26T02:23:01.142 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-04-26T02:23:01.143 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T02:23:01.144 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-04-26T02:23:01.145 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "[v1:172.21.15.177:6789],[v1:172.21.15.177:6790],[v1:172.21.15.177:6791]" 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: mon client hunt interval max multiple = 2 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr beacon grace = 90 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: ms bind msgr2 = False 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 1000 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 13088462-cff2-40f2-960d-bd23a2ee33c5 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-04-26T02:23:01.146 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-04-26T02:23:01.147 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.148 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression algorithm = lz4 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression mode = aggressive 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 20 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 20 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-04-26T02:23:01.149 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-04-26T02:23:01.150 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 10 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-04-26T02:23:01.151 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T02:23:01.152 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T02:23:01.153 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-04-26T02:23:01.154 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-04-26T02:23:01.154 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-04-26T02:23:01.155 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T02:23:01.155 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T02:23:01.245 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T02:23:01.245 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:01.245 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T02:23:01.250 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:01.250 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T02:23:01.296 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:23:01.296 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:01.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T02:23:01.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T02:23:01.362 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:01.362 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:01.409 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T02:23:01.409 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T02:23:01.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T02:23:01.494 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T02:23:01.496 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T02:23:01.497 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T02:23:01.497 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T02:23:01.497 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-26T02:23:01.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T02:23:01.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:23:01.557 INFO:tasks.ceph:role: osd.0 2024-04-26T02:23:01.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:23:01.558 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:23:01.620 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:23:01.621 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:23:01.621 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:23:02.070 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-26T02:23:02.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-26T02:23:02.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T02:23:02.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T02:23:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:23:02.184 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:23:02.184 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T02:23:02.241 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:23:02.241 INFO:tasks.ceph:role: osd.1 2024-04-26T02:23:02.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:23:02.242 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:23:02.304 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:23:02.305 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:23:02.305 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:23:02.305 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:23:02.781 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-26T02:23:02.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-26T02:23:02.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T02:23:02.933 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T02:23:02.944 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:23:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:23:02.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T02:23:03.000 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:23:03.001 INFO:tasks.ceph:role: osd.2 2024-04-26T02:23:03.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:23:03.001 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:23:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:23:03.538 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-26T02:23:03.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-26T02:23:03.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T02:23:03.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T02:23:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:23:03.688 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:23:03.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T02:23:03.744 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:23:03.744 INFO:tasks.ceph:role: osd.3 2024-04-26T02:23:03.745 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:23:03.745 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T02:23:03.808 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:23:03.809 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:23:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:23:03.810 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:23:04.233 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-26T02:23:04.235 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-26T02:23:04.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-26T02:23:04.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T02:23:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:23:04.373 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:23:04.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:04.484 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.484+0000 7f14597b77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T02:23:04.490 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.488+0000 7f14597b77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T02:23:04.490 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.488+0000 7f14597b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:23:04.490 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.488+0000 7f14597b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:23:04.490 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.488+0000 7f14597b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:23:04.492 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:04.492+0000 7f14597b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T02:23:07.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T02:23:07.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:07.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T02:23:07.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T02:23:07.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:23:07.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:23:07.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:23:07.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:07.572+0000 7f02c70587c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T02:23:10.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T02:23:10.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:10.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.684+0000 7f88c75577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T02:23:10.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.684+0000 7f88c75577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T02:23:10.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.684+0000 7f88c75577c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:23:10.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.684+0000 7f88c75577c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:23:10.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.684+0000 7f88c75577c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:23:10.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:10.688+0000 7f88c75577c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T02:23:13.644 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T02:23:13.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:13.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T02:23:13.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T02:23:13.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:23:13.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:23:13.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:23:13.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-26T02:23:13.780+0000 7fbc4e50b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T02:23:16.754 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T02:23:16.810 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T02:23:16.810 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:16.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T02:23:16.863 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:16.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T02:23:16.917 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:16.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T02:23:16.972 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:16.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T02:23:17.032 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:17.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T02:23:17.087 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T02:23:17.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAFECtm/mORHRAAdsyn0rN2JrKLs+hZY9Y8fg== 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAIECtmNd4DHRAAmRyHKu7iHhbyfT+WEN+4xA== 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQALECtm0vlKIhAA2+euegEzm0GVcx5Z1Uvuww== 2024-04-26T02:23:17.138 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-04-26T02:23:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAOECtm2bAbKRAAuHPSxKlxrN89CPJt5MZJlA== 2024-04-26T02:23:17.139 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-04-26T02:23:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQARECtmPJS0LhAAeu5zOJw9I7ky6wbPAttGmg== 2024-04-26T02:23:17.139 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T02:23:17.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:23:17.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:23:17.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:23:17.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:23:17.524 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T02:23:17.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T02:23:17.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:23:17.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T02:23:17.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T02:23:17.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:23:17.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T02:23:17.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T02:23:17.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:23:17.982 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T02:23:18.036 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:23:18.084 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T02:23:18.085 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T02:23:18.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T02:23:18.129 INFO:tasks.ceph.mon.a:Started 2024-04-26T02:23:18.130 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T02:23:18.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T02:23:18.132 INFO:tasks.ceph.mon.b:Started 2024-04-26T02:23:18.132 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T02:23:18.133 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T02:23:18.135 INFO:tasks.ceph.mon.c:Started 2024-04-26T02:23:18.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph config set mgr mgr_pool false --force 2024-04-26T02:23:23.608 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T02:23:23.608 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T02:23:23.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T02:23:23.611 INFO:tasks.ceph.mgr.x:Started 2024-04-26T02:23:23.611 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T02:23:23.611 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T02:23:23.805 INFO:teuthology.orchestra.run.smithi177.stderr:adjusted tunables profile to default 2024-04-26T02:23:23.823 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T02:23:23.823 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T02:23:23.823 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:23.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T02:23:23.833 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:23.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T02:23:23.883 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:23.880+0000 7fe5b6664180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T02:23:23.885 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:23.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T02:23:23.939 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-26T02:23:23.940 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T02:23:23.995 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 77e13700-dc07-44ef-92cd-f9e7b3b12087 0 2024-04-26T02:23:24.233 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-04-26T02:23:24.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new bc6b2040-3b7c-410f-b7da-a33b88edf242 1 2024-04-26T02:23:24.352 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:24.352+0000 7fe5b6664180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T02:23:24.443 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-26T02:23:24.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 1bcd23e3-1de4-494e-b73d-51c88edda7a5 2 2024-04-26T02:23:24.557 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:24.556+0000 7fe5b6664180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T02:23:24.649 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-04-26T02:23:24.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 51cee477-d535-4ba9-bee6-4288e817a9d2 3 2024-04-26T02:23:24.854 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-04-26T02:23:24.870 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T02:23:24.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T02:23:24.872 INFO:tasks.ceph.osd.0:Started 2024-04-26T02:23:24.872 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T02:23:24.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T02:23:24.874 INFO:tasks.ceph.osd.1:Started 2024-04-26T02:23:24.874 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T02:23:24.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T02:23:24.876 INFO:tasks.ceph.osd.2:Started 2024-04-26T02:23:24.876 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T02:23:24.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T02:23:24.878 INFO:tasks.ceph.osd.3:Started 2024-04-26T02:23:24.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:25.074 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:25.074 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":6,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:24.853353+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:25.090 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T02:23:25.090 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T02:23:25.170 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.168+0000 7fe5b6664180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T02:23:25.229 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.228+0000 7fe5b6664180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T02:23:25.282 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.280+0000 7fe5b6664180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T02:23:25.359 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.356+0000 7fe5b6664180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T02:23:25.410 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.408+0000 7fe5b6664180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T02:23:25.485 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-04-26T02:23:25.484+0000 7fef624187c0 -1 Falling back to public interface 2024-04-26T02:23:25.486 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:25.484+0000 7f6d131d77c0 -1 Falling back to public interface 2024-04-26T02:23:25.487 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.484+0000 7fe5b6664180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T02:23:25.493 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:25.500 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:25.500+0000 7fde9d7027c0 -1 Falling back to public interface 2024-04-26T02:23:25.516 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:25.516+0000 7f4e646797c0 -1 Falling back to public interface 2024-04-26T02:23:25.546 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.544+0000 7fe5b6664180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T02:23:25.666 INFO:teuthology.misc.health.smithi177.stdout: 2024-04-26T02:23:25.666 INFO:teuthology.misc.health.smithi177.stdout:{"epoch":6,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:24.853353+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:25.682 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-26T02:23:25.700 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.700+0000 7fe5b6664180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T02:23:25.761 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:25.760+0000 7fe5b6664180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T02:23:26.369 INFO:tasks.ceph.mgr.x.smithi177.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T02:23:26.369 INFO:tasks.ceph.mgr.x.smithi177.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T02:23:26.369 INFO:tasks.ceph.mgr.x.smithi177.stderr: from numpy import show_config as show_numpy_config 2024-04-26T02:23:26.439 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:26.436+0000 7fe5b6664180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T02:23:26.621 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:26.620+0000 7fe5b6664180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:23:26.793 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:26.792+0000 7fe5b6664180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T02:23:26.862 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:26.860+0000 7fe5b6664180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T02:23:27.009 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:27.008+0000 7fe5b6664180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:23:27.661 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:27.660+0000 7fe5b6664180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T02:23:27.744 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:27.744+0000 7fe5b6664180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T02:23:27.933 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:27.932+0000 7fe5b6664180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T02:23:28.008 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:28.008+0000 7fe5b6664180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T02:23:28.303 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:28.300+0000 7fe5b6664180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T02:23:28.375 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:28.372+0000 7fe5b6664180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T02:23:29.143 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:29.140+0000 7fde9d7027c0 -1 osd.1 0 log_to_monitors true 2024-04-26T02:23:29.151 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-04-26T02:23:29.148+0000 7fef624187c0 -1 osd.2 0 log_to_monitors true 2024-04-26T02:23:29.158 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:29.156+0000 7f6d131d77c0 -1 osd.3 0 log_to_monitors true 2024-04-26T02:23:29.335 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:29.332+0000 7f4e646797c0 -1 osd.0 0 log_to_monitors true 2024-04-26T02:23:29.419 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:29.416+0000 7fe55dab1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T02:23:29.421 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-26T02:23:29.420+0000 7fe55dab1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T02:23:30.501 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-04-26T02:23:30.500+0000 7fef5e3c1640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T02:23:30.510 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-04-26T02:23:30.508+0000 7fef5a1fe640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:23:31.453 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:31.452+0000 7f6d0f180640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T02:23:31.457 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:31.456+0000 7f6d0f180640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T02:23:31.457 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:31.456+0000 7f6d0f180640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T02:23:31.457 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:31.456+0000 7f6d0f992640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T02:23:31.460 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:23:31.460+0000 7f6d06fb5640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:23:31.469 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:31.468+0000 7fde99ebd640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T02:23:31.470 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:31.468+0000 7f4e60622640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T02:23:31.472 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:31.472+0000 7fde99ebd640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T02:23:31.473 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:31.472+0000 7fde99ebd640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T02:23:31.473 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:31.472+0000 7fde99ebd640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T02:23:31.473 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:23:31.472+0000 7fde914e0640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:23:31.474 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:31.472+0000 7f4e60622640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T02:23:31.474 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:31.472+0000 7f4e60622640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T02:23:31.474 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:31.472+0000 7f4e60622640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T02:23:31.477 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:23:31.476+0000 7f4e58457640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:23:32.085 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:32.373 INFO:teuthology.misc.health.smithi177.stdout: 2024-04-26T02:23:32.374 INFO:teuthology.misc.health.smithi177.stdout:{"epoch":9,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:31.508511+0000","last_up_change":"2024-04-26T02:23:31.508511+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":775556026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":775556026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":775556026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":775556026}]},"public_addr":"172.21.15.177:6812/775556026","cluster_addr":"172.21.15.177:6813/775556026","heartbeat_back_addr":"172.21.15.177:6815/775556026","heartbeat_front_addr":"172.21.15.177:6814/775556026","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:32.390 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-26T02:23:32.390 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T02:23:32.390 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T02:23:32.390 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T02:23:32.391 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T02:23:32.391 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T02:23:32.706 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:32.721 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":900423765}]},"active_addr":"172.21.15.177:6816/900423765","active_change":"2024-04-26T02:23:28.400031+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":863634891}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1467006646}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3649844867}]}]} 2024-04-26T02:23:32.724 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T02:23:32.724 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T02:23:32.724 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:32.996 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:32.996 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":10,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:32.514193+0000","last_up_change":"2024-04-26T02:23:31.508511+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":775556026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":775556026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":775556026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":775556026}]},"public_addr":"172.21.15.177:6812/775556026","cluster_addr":"172.21.15.177:6813/775556026","heartbeat_back_addr":"172.21.15.177:6815/775556026","heartbeat_front_addr":"172.21.15.177:6814/775556026","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:33.012 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T02:23:33.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:33.278 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:33.278 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":10,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:32.514193+0000","last_up_change":"2024-04-26T02:23:31.508511+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":775556026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":775556026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":775556026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":775556026}]},"public_addr":"172.21.15.177:6812/775556026","cluster_addr":"172.21.15.177:6813/775556026","heartbeat_back_addr":"172.21.15.177:6815/775556026","heartbeat_front_addr":"172.21.15.177:6814/775556026","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:33.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:23:33.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T02:23:33.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T02:23:33.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T02:23:33.454 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:33.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:23:33.461 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:33.461 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:23:33.466 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:33.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:23:33.468 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:33.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:23:33.739 INFO:teuthology.orchestra.run.smithi177.stdout:38654705665 2024-04-26T02:23:33.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2024-04-26T02:23:33.779 INFO:teuthology.orchestra.run.smithi177.stdout:38654705665 2024-04-26T02:23:33.795 INFO:teuthology.orchestra.run.smithi177.stdout:38654705665 2024-04-26T02:23:33.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.3 2024-04-26T02:23:33.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.0 2024-04-26T02:23:33.834 INFO:teuthology.orchestra.run.smithi177.stdout:38654705665 2024-04-26T02:23:33.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2024-04-26T02:23:34.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:23:34.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:23:34.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:23:34.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:23:35.100 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:35.110 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:35.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.3 2024-04-26T02:23:35.118 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:35.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2024-04-26T02:23:35.128 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:35.143 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:35.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.0 2024-04-26T02:23:35.159 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:35.166 INFO:teuthology.orchestra.run.smithi177.stdout:38654705666 2024-04-26T02:23:35.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2024-04-26T02:23:35.181 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:35.181 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T02:23:35.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:35.491 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:35.491 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:35.507 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-26T02:23:34.409432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108436,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378924,"statfs":{"total":386547056640,"available":386436018176,"internally_reserved":0,"allocated":352256,"data_stored":72107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27100,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":81920,"data_stored":16172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-26T02:23:35.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:35.769 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:35.769 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:35.786 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-26T02:23:34.409432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108436,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378924,"statfs":{"total":386547056640,"available":386436018176,"internally_reserved":0,"allocated":352256,"data_stored":72107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27100,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":81920,"data_stored":16172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-26T02:23:35.787 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T02:23:35.787 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T02:23:35.787 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T02:23:35.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T02:23:36.078 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:36.079 INFO:teuthology.orchestra.run.smithi177.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T02:23:36.095 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T02:23:36.095 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-04-26T02:23:36.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:23:36.393 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:36.394 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":10,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:23:32.514193+0000","last_up_change":"2024-04-26T02:23:31.508511+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":775556026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":775556026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":775556026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":775556026}]},"public_addr":"172.21.15.177:6812/775556026","cluster_addr":"172.21.15.177:6813/775556026","heartbeat_back_addr":"172.21.15.177:6815/775556026","heartbeat_front_addr":"172.21.15.177:6814/775556026","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:23:36.410 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-04-26T02:23:36.410 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-04-26T02:23:36.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:37.526 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:37.552 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T02:23:36.409907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108436,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378924,"statfs":{"total":386547056640,"available":386436018176,"internally_reserved":0,"allocated":352256,"data_stored":72107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27100,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":81920,"data_stored":16172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-26T02:23:37.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:37.832 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:37.848 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T02:23:36.409907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108436,"kb_used_data":344,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378924,"statfs":{"total":386547056640,"available":386436018176,"internally_reserved":0,"allocated":352256,"data_stored":72107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27100,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":81920,"data_stored":16172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-26T02:23:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-04-26T02:23:37.849 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-04-26T02:23:38.571 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-04-26T02:23:38.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-04-26T02:23:40.579 INFO:teuthology.orchestra.run.smithi177.stderr:pool 'unique_pool_0' created 2024-04-26T02:23:40.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-04-26T02:23:41.582 INFO:teuthology.orchestra.run.smithi177.stderr:set pool 1 min_size to 2 2024-04-26T02:23:41.614 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-26T02:23:42.573 INFO:teuthology.orchestra.run.smithi177.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-26T02:23:43.611 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-04-26T02:23:43.611 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-04-26T02:23:43.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:23:43.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T02:23:43.817 INFO:teuthology.orchestra.run.smithi177.stdout:38654705669 2024-04-26T02:23:43.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:23:43.821 INFO:teuthology.orchestra.run.smithi177.stdout:38654705669 2024-04-26T02:23:43.822 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:23:44.095 INFO:teuthology.orchestra.run.smithi177.stdout:38654705667 2024-04-26T02:23:44.111 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705667 for osd.1 2024-04-26T02:23:44.140 INFO:teuthology.orchestra.run.smithi177.stdout:38654705667 2024-04-26T02:23:44.158 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705667 for osd.0 2024-04-26T02:23:45.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:23:45.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:23:45.402 INFO:teuthology.orchestra.run.smithi177.stdout:38654705669 2024-04-26T02:23:45.419 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705669 for osd.1 2024-04-26T02:23:45.419 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:45.436 INFO:teuthology.orchestra.run.smithi177.stdout:38654705669 2024-04-26T02:23:45.453 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705669 for osd.0 2024-04-26T02:23:45.453 DEBUG:teuthology.parallel:result is None 2024-04-26T02:23:45.453 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-04-26T02:23:45.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:45.769 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:45.769 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:45.785 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-26T02:23:44.411748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108596,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378764,"statfs":{"total":386547056640,"available":386435854336,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593492+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593492+0000","last_undegraded":"2024-04-26T02:23:41.593492+0000","last_fullsized":"2024-04-26T02:23:41.593492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663539+0000","last_change":"2024-04-26T02:23:42.616113+0000","last_active":"2024-04-26T02:23:43.663539+0000","last_peered":"2024-04-26T02:23:43.663539+0000","last_clean":"2024-04-26T02:23:43.663539+0000","last_became_active":"2024-04-26T02:23:42.614337+0000","last_became_peered":"2024-04-26T02:23:42.614337+0000","last_unstale":"2024-04-26T02:23:43.663539+0000","last_undegraded":"2024-04-26T02:23:43.663539+0000","last_fullsized":"2024-04-26T02:23:43.663539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:42:50.781694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588746+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588746+0000","last_undegraded":"2024-04-26T02:23:41.588746+0000","last_fullsized":"2024-04-26T02:23:41.588746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663078+0000","last_change":"2024-04-26T02:23:42.617820+0000","last_active":"2024-04-26T02:23:43.663078+0000","last_peered":"2024-04-26T02:23:43.663078+0000","last_clean":"2024-04-26T02:23:43.663078+0000","last_became_active":"2024-04-26T02:23:42.617520+0000","last_became_peered":"2024-04-26T02:23:42.617520+0000","last_unstale":"2024-04-26T02:23:43.663078+0000","last_undegraded":"2024-04-26T02:23:43.663078+0000","last_fullsized":"2024-04-26T02:23:43.663078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:50:00.523082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590090+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590090+0000","last_undegraded":"2024-04-26T02:23:41.590090+0000","last_fullsized":"2024-04-26T02:23:41.590090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590044+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590044+0000","last_undegraded":"2024-04-26T02:23:41.590044+0000","last_fullsized":"2024-04-26T02:23:41.590044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.616952+0000","last_change":"2024-04-26T02:23:42.613541+0000","last_active":"2024-04-26T02:23:43.616952+0000","last_peered":"2024-04-26T02:23:43.616952+0000","last_clean":"2024-04-26T02:23:43.616952+0000","last_became_active":"2024-04-26T02:23:42.612975+0000","last_became_peered":"2024-04-26T02:23:42.612975+0000","last_unstale":"2024-04-26T02:23:43.616952+0000","last_undegraded":"2024-04-26T02:23:43.616952+0000","last_fullsized":"2024-04-26T02:23:43.616952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:52:20.904007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663364+0000","last_change":"2024-04-26T02:23:42.606688+0000","last_active":"2024-04-26T02:23:43.663364+0000","last_peered":"2024-04-26T02:23:43.663364+0000","last_clean":"2024-04-26T02:23:43.663364+0000","last_became_active":"2024-04-26T02:23:42.606239+0000","last_became_peered":"2024-04-26T02:23:42.606239+0000","last_unstale":"2024-04-26T02:23:43.663364+0000","last_undegraded":"2024-04-26T02:23:43.663364+0000","last_fullsized":"2024-04-26T02:23:43.663364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:16:06.174520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.591509+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.591509+0000","last_undegraded":"2024-04-26T02:23:41.591509+0000","last_fullsized":"2024-04-26T02:23:41.591509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588091+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588091+0000","last_undegraded":"2024-04-26T02:23:41.588091+0000","last_fullsized":"2024-04-26T02:23:41.588091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593396+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593396+0000","last_undegraded":"2024-04-26T02:23:41.593396+0000","last_fullsized":"2024-04-26T02:23:41.593396+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663808+0000","last_change":"2024-04-26T02:23:42.607673+0000","last_active":"2024-04-26T02:23:43.663808+0000","last_peered":"2024-04-26T02:23:43.663808+0000","last_clean":"2024-04-26T02:23:43.663808+0000","last_became_active":"2024-04-26T02:23:42.607271+0000","last_became_peered":"2024-04-26T02:23:42.607271+0000","last_unstale":"2024-04-26T02:23:43.663808+0000","last_undegraded":"2024-04-26T02:23:43.663808+0000","last_fullsized":"2024-04-26T02:23:43.663808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:59.594516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663025+0000","last_change":"2024-04-26T02:23:42.617669+0000","last_active":"2024-04-26T02:23:43.663025+0000","last_peered":"2024-04-26T02:23:43.663025+0000","last_clean":"2024-04-26T02:23:43.663025+0000","last_became_active":"2024-04-26T02:23:42.617075+0000","last_became_peered":"2024-04-26T02:23:42.617075+0000","last_unstale":"2024-04-26T02:23:43.663025+0000","last_undegraded":"2024-04-26T02:23:43.663025+0000","last_fullsized":"2024-04-26T02:23:43.663025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:48.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663323+0000","last_change":"2024-04-26T02:23:42.607210+0000","last_active":"2024-04-26T02:23:43.663323+0000","last_peered":"2024-04-26T02:23:43.663323+0000","last_clean":"2024-04-26T02:23:43.663323+0000","last_became_active":"2024-04-26T02:23:42.606647+0000","last_became_peered":"2024-04-26T02:23:42.606647+0000","last_unstale":"2024-04-26T02:23:43.663323+0000","last_undegraded":"2024-04-26T02:23:43.663323+0000","last_fullsized":"2024-04-26T02:23:43.663323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:03:21.145208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663255+0000","last_change":"2024-04-26T02:23:42.615863+0000","last_active":"2024-04-26T02:23:43.663255+0000","last_peered":"2024-04-26T02:23:43.663255+0000","last_clean":"2024-04-26T02:23:43.663255+0000","last_became_active":"2024-04-26T02:23:42.614960+0000","last_became_peered":"2024-04-26T02:23:42.614960+0000","last_unstale":"2024-04-26T02:23:43.663255+0000","last_undegraded":"2024-04-26T02:23:43.663255+0000","last_fullsized":"2024-04-26T02:23:43.663255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:45.378184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663130+0000","last_change":"2024-04-26T02:23:42.607875+0000","last_active":"2024-04-26T02:23:43.663130+0000","last_peered":"2024-04-26T02:23:43.663130+0000","last_clean":"2024-04-26T02:23:43.663130+0000","last_became_active":"2024-04-26T02:23:42.607094+0000","last_became_peered":"2024-04-26T02:23:42.607094+0000","last_unstale":"2024-04-26T02:23:43.663130+0000","last_undegraded":"2024-04-26T02:23:43.663130+0000","last_fullsized":"2024-04-26T02:23:43.663130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:50:35.730183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:23:45.786 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:46.052 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:46.053 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:46.069 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-26T02:23:44.411748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108596,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378764,"statfs":{"total":386547056640,"available":386435854336,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593492+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593492+0000","last_undegraded":"2024-04-26T02:23:41.593492+0000","last_fullsized":"2024-04-26T02:23:41.593492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663539+0000","last_change":"2024-04-26T02:23:42.616113+0000","last_active":"2024-04-26T02:23:43.663539+0000","last_peered":"2024-04-26T02:23:43.663539+0000","last_clean":"2024-04-26T02:23:43.663539+0000","last_became_active":"2024-04-26T02:23:42.614337+0000","last_became_peered":"2024-04-26T02:23:42.614337+0000","last_unstale":"2024-04-26T02:23:43.663539+0000","last_undegraded":"2024-04-26T02:23:43.663539+0000","last_fullsized":"2024-04-26T02:23:43.663539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:42:50.781694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588746+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588746+0000","last_undegraded":"2024-04-26T02:23:41.588746+0000","last_fullsized":"2024-04-26T02:23:41.588746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663078+0000","last_change":"2024-04-26T02:23:42.617820+0000","last_active":"2024-04-26T02:23:43.663078+0000","last_peered":"2024-04-26T02:23:43.663078+0000","last_clean":"2024-04-26T02:23:43.663078+0000","last_became_active":"2024-04-26T02:23:42.617520+0000","last_became_peered":"2024-04-26T02:23:42.617520+0000","last_unstale":"2024-04-26T02:23:43.663078+0000","last_undegraded":"2024-04-26T02:23:43.663078+0000","last_fullsized":"2024-04-26T02:23:43.663078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:50:00.523082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590090+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590090+0000","last_undegraded":"2024-04-26T02:23:41.590090+0000","last_fullsized":"2024-04-26T02:23:41.590090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590044+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590044+0000","last_undegraded":"2024-04-26T02:23:41.590044+0000","last_fullsized":"2024-04-26T02:23:41.590044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.616952+0000","last_change":"2024-04-26T02:23:42.613541+0000","last_active":"2024-04-26T02:23:43.616952+0000","last_peered":"2024-04-26T02:23:43.616952+0000","last_clean":"2024-04-26T02:23:43.616952+0000","last_became_active":"2024-04-26T02:23:42.612975+0000","last_became_peered":"2024-04-26T02:23:42.612975+0000","last_unstale":"2024-04-26T02:23:43.616952+0000","last_undegraded":"2024-04-26T02:23:43.616952+0000","last_fullsized":"2024-04-26T02:23:43.616952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:52:20.904007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663364+0000","last_change":"2024-04-26T02:23:42.606688+0000","last_active":"2024-04-26T02:23:43.663364+0000","last_peered":"2024-04-26T02:23:43.663364+0000","last_clean":"2024-04-26T02:23:43.663364+0000","last_became_active":"2024-04-26T02:23:42.606239+0000","last_became_peered":"2024-04-26T02:23:42.606239+0000","last_unstale":"2024-04-26T02:23:43.663364+0000","last_undegraded":"2024-04-26T02:23:43.663364+0000","last_fullsized":"2024-04-26T02:23:43.663364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:16:06.174520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.591509+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.591509+0000","last_undegraded":"2024-04-26T02:23:41.591509+0000","last_fullsized":"2024-04-26T02:23:41.591509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588091+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588091+0000","last_undegraded":"2024-04-26T02:23:41.588091+0000","last_fullsized":"2024-04-26T02:23:41.588091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593396+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593396+0000","last_undegraded":"2024-04-26T02:23:41.593396+0000","last_fullsized":"2024-04-26T02:23:41.593396+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663808+0000","last_change":"2024-04-26T02:23:42.607673+0000","last_active":"2024-04-26T02:23:43.663808+0000","last_peered":"2024-04-26T02:23:43.663808+0000","last_clean":"2024-04-26T02:23:43.663808+0000","last_became_active":"2024-04-26T02:23:42.607271+0000","last_became_peered":"2024-04-26T02:23:42.607271+0000","last_unstale":"2024-04-26T02:23:43.663808+0000","last_undegraded":"2024-04-26T02:23:43.663808+0000","last_fullsized":"2024-04-26T02:23:43.663808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:59.594516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663025+0000","last_change":"2024-04-26T02:23:42.617669+0000","last_active":"2024-04-26T02:23:43.663025+0000","last_peered":"2024-04-26T02:23:43.663025+0000","last_clean":"2024-04-26T02:23:43.663025+0000","last_became_active":"2024-04-26T02:23:42.617075+0000","last_became_peered":"2024-04-26T02:23:42.617075+0000","last_unstale":"2024-04-26T02:23:43.663025+0000","last_undegraded":"2024-04-26T02:23:43.663025+0000","last_fullsized":"2024-04-26T02:23:43.663025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:48.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663323+0000","last_change":"2024-04-26T02:23:42.607210+0000","last_active":"2024-04-26T02:23:43.663323+0000","last_peered":"2024-04-26T02:23:43.663323+0000","last_clean":"2024-04-26T02:23:43.663323+0000","last_became_active":"2024-04-26T02:23:42.606647+0000","last_became_peered":"2024-04-26T02:23:42.606647+0000","last_unstale":"2024-04-26T02:23:43.663323+0000","last_undegraded":"2024-04-26T02:23:43.663323+0000","last_fullsized":"2024-04-26T02:23:43.663323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:03:21.145208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663255+0000","last_change":"2024-04-26T02:23:42.615863+0000","last_active":"2024-04-26T02:23:43.663255+0000","last_peered":"2024-04-26T02:23:43.663255+0000","last_clean":"2024-04-26T02:23:43.663255+0000","last_became_active":"2024-04-26T02:23:42.614960+0000","last_became_peered":"2024-04-26T02:23:42.614960+0000","last_unstale":"2024-04-26T02:23:43.663255+0000","last_undegraded":"2024-04-26T02:23:43.663255+0000","last_fullsized":"2024-04-26T02:23:43.663255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:45.378184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663130+0000","last_change":"2024-04-26T02:23:42.607875+0000","last_active":"2024-04-26T02:23:43.663130+0000","last_peered":"2024-04-26T02:23:43.663130+0000","last_clean":"2024-04-26T02:23:43.663130+0000","last_became_active":"2024-04-26T02:23:42.607094+0000","last_became_peered":"2024-04-26T02:23:42.607094+0000","last_unstale":"2024-04-26T02:23:43.663130+0000","last_undegraded":"2024-04-26T02:23:43.663130+0000","last_fullsized":"2024-04-26T02:23:43.663130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:50:35.730183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:23:46.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:46.341 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:46.342 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:46.357 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-26T02:23:44.411748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108596,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378764,"statfs":{"total":386547056640,"available":386435854336,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593492+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593492+0000","last_undegraded":"2024-04-26T02:23:41.593492+0000","last_fullsized":"2024-04-26T02:23:41.593492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663539+0000","last_change":"2024-04-26T02:23:42.616113+0000","last_active":"2024-04-26T02:23:43.663539+0000","last_peered":"2024-04-26T02:23:43.663539+0000","last_clean":"2024-04-26T02:23:43.663539+0000","last_became_active":"2024-04-26T02:23:42.614337+0000","last_became_peered":"2024-04-26T02:23:42.614337+0000","last_unstale":"2024-04-26T02:23:43.663539+0000","last_undegraded":"2024-04-26T02:23:43.663539+0000","last_fullsized":"2024-04-26T02:23:43.663539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:42:50.781694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588746+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588746+0000","last_undegraded":"2024-04-26T02:23:41.588746+0000","last_fullsized":"2024-04-26T02:23:41.588746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663078+0000","last_change":"2024-04-26T02:23:42.617820+0000","last_active":"2024-04-26T02:23:43.663078+0000","last_peered":"2024-04-26T02:23:43.663078+0000","last_clean":"2024-04-26T02:23:43.663078+0000","last_became_active":"2024-04-26T02:23:42.617520+0000","last_became_peered":"2024-04-26T02:23:42.617520+0000","last_unstale":"2024-04-26T02:23:43.663078+0000","last_undegraded":"2024-04-26T02:23:43.663078+0000","last_fullsized":"2024-04-26T02:23:43.663078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:50:00.523082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590090+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590090+0000","last_undegraded":"2024-04-26T02:23:41.590090+0000","last_fullsized":"2024-04-26T02:23:41.590090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.590044+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.590044+0000","last_undegraded":"2024-04-26T02:23:41.590044+0000","last_fullsized":"2024-04-26T02:23:41.590044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.616952+0000","last_change":"2024-04-26T02:23:42.613541+0000","last_active":"2024-04-26T02:23:43.616952+0000","last_peered":"2024-04-26T02:23:43.616952+0000","last_clean":"2024-04-26T02:23:43.616952+0000","last_became_active":"2024-04-26T02:23:42.612975+0000","last_became_peered":"2024-04-26T02:23:42.612975+0000","last_unstale":"2024-04-26T02:23:43.616952+0000","last_undegraded":"2024-04-26T02:23:43.616952+0000","last_fullsized":"2024-04-26T02:23:43.616952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:52:20.904007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663364+0000","last_change":"2024-04-26T02:23:42.606688+0000","last_active":"2024-04-26T02:23:43.663364+0000","last_peered":"2024-04-26T02:23:43.663364+0000","last_clean":"2024-04-26T02:23:43.663364+0000","last_became_active":"2024-04-26T02:23:42.606239+0000","last_became_peered":"2024-04-26T02:23:42.606239+0000","last_unstale":"2024-04-26T02:23:43.663364+0000","last_undegraded":"2024-04-26T02:23:43.663364+0000","last_fullsized":"2024-04-26T02:23:43.663364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:16:06.174520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.591509+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.591509+0000","last_undegraded":"2024-04-26T02:23:41.591509+0000","last_fullsized":"2024-04-26T02:23:41.591509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.588091+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.588091+0000","last_undegraded":"2024-04-26T02:23:41.588091+0000","last_fullsized":"2024-04-26T02:23:41.588091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-04-26T02:23:41.593396+0000","last_change":"2024-04-26T02:23:40.559762+0000","last_active":"2024-04-26T02:23:40.559762+0000","last_peered":"2024-04-26T02:23:40.559762+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:23:41.593396+0000","last_undegraded":"2024-04-26T02:23:41.593396+0000","last_fullsized":"2024-04-26T02:23:41.593396+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663808+0000","last_change":"2024-04-26T02:23:42.607673+0000","last_active":"2024-04-26T02:23:43.663808+0000","last_peered":"2024-04-26T02:23:43.663808+0000","last_clean":"2024-04-26T02:23:43.663808+0000","last_became_active":"2024-04-26T02:23:42.607271+0000","last_became_peered":"2024-04-26T02:23:42.607271+0000","last_unstale":"2024-04-26T02:23:43.663808+0000","last_undegraded":"2024-04-26T02:23:43.663808+0000","last_fullsized":"2024-04-26T02:23:43.663808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:59.594516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663025+0000","last_change":"2024-04-26T02:23:42.617669+0000","last_active":"2024-04-26T02:23:43.663025+0000","last_peered":"2024-04-26T02:23:43.663025+0000","last_clean":"2024-04-26T02:23:43.663025+0000","last_became_active":"2024-04-26T02:23:42.617075+0000","last_became_peered":"2024-04-26T02:23:42.617075+0000","last_unstale":"2024-04-26T02:23:43.663025+0000","last_undegraded":"2024-04-26T02:23:43.663025+0000","last_fullsized":"2024-04-26T02:23:43.663025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:48.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663323+0000","last_change":"2024-04-26T02:23:42.607210+0000","last_active":"2024-04-26T02:23:43.663323+0000","last_peered":"2024-04-26T02:23:43.663323+0000","last_clean":"2024-04-26T02:23:43.663323+0000","last_became_active":"2024-04-26T02:23:42.606647+0000","last_became_peered":"2024-04-26T02:23:42.606647+0000","last_unstale":"2024-04-26T02:23:43.663323+0000","last_undegraded":"2024-04-26T02:23:43.663323+0000","last_fullsized":"2024-04-26T02:23:43.663323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:03:21.145208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663255+0000","last_change":"2024-04-26T02:23:42.615863+0000","last_active":"2024-04-26T02:23:43.663255+0000","last_peered":"2024-04-26T02:23:43.663255+0000","last_clean":"2024-04-26T02:23:43.663255+0000","last_became_active":"2024-04-26T02:23:42.614960+0000","last_became_peered":"2024-04-26T02:23:42.614960+0000","last_unstale":"2024-04-26T02:23:43.663255+0000","last_undegraded":"2024-04-26T02:23:43.663255+0000","last_fullsized":"2024-04-26T02:23:43.663255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:45.378184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663130+0000","last_change":"2024-04-26T02:23:42.607875+0000","last_active":"2024-04-26T02:23:43.663130+0000","last_peered":"2024-04-26T02:23:43.663130+0000","last_clean":"2024-04-26T02:23:43.663130+0000","last_became_active":"2024-04-26T02:23:42.607094+0000","last_became_peered":"2024-04-26T02:23:42.607094+0000","last_unstale":"2024-04-26T02:23:43.663130+0000","last_undegraded":"2024-04-26T02:23:43.663130+0000","last_fullsized":"2024-04-26T02:23:43.663130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:50:35.730183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:23:49.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:23:49.663 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:23:49.663 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:23:49.680 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-26T02:23:48.412745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108648,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378712,"statfs":{"total":386547056640,"available":386435801088,"internally_reserved":0,"allocated":573440,"data_stored":170680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.660885+0000","last_change":"2024-04-26T02:23:42.609546+0000","last_active":"2024-04-26T02:23:43.660885+0000","last_peered":"2024-04-26T02:23:43.660885+0000","last_clean":"2024-04-26T02:23:43.660885+0000","last_became_active":"2024-04-26T02:23:42.608415+0000","last_became_peered":"2024-04-26T02:23:42.608415+0000","last_unstale":"2024-04-26T02:23:43.660885+0000","last_undegraded":"2024-04-26T02:23:43.660885+0000","last_fullsized":"2024-04-26T02:23:43.660885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:10.316465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663539+0000","last_change":"2024-04-26T02:23:42.616113+0000","last_active":"2024-04-26T02:23:43.663539+0000","last_peered":"2024-04-26T02:23:43.663539+0000","last_clean":"2024-04-26T02:23:43.663539+0000","last_became_active":"2024-04-26T02:23:42.614337+0000","last_became_peered":"2024-04-26T02:23:42.614337+0000","last_unstale":"2024-04-26T02:23:43.663539+0000","last_undegraded":"2024-04-26T02:23:43.663539+0000","last_fullsized":"2024-04-26T02:23:43.663539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:42:50.781694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.618075+0000","last_change":"2024-04-26T02:23:42.603680+0000","last_active":"2024-04-26T02:23:43.618075+0000","last_peered":"2024-04-26T02:23:43.618075+0000","last_clean":"2024-04-26T02:23:43.618075+0000","last_became_active":"2024-04-26T02:23:42.603002+0000","last_became_peered":"2024-04-26T02:23:42.603002+0000","last_unstale":"2024-04-26T02:23:43.618075+0000","last_undegraded":"2024-04-26T02:23:43.618075+0000","last_fullsized":"2024-04-26T02:23:43.618075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:49:39.586747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663078+0000","last_change":"2024-04-26T02:23:42.617820+0000","last_active":"2024-04-26T02:23:43.663078+0000","last_peered":"2024-04-26T02:23:43.663078+0000","last_clean":"2024-04-26T02:23:43.663078+0000","last_became_active":"2024-04-26T02:23:42.617520+0000","last_became_peered":"2024-04-26T02:23:42.617520+0000","last_unstale":"2024-04-26T02:23:43.663078+0000","last_undegraded":"2024-04-26T02:23:43.663078+0000","last_fullsized":"2024-04-26T02:23:43.663078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:50:00.523082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.660568+0000","last_change":"2024-04-26T02:23:42.615656+0000","last_active":"2024-04-26T02:23:43.660568+0000","last_peered":"2024-04-26T02:23:43.660568+0000","last_clean":"2024-04-26T02:23:43.660568+0000","last_became_active":"2024-04-26T02:23:42.615166+0000","last_became_peered":"2024-04-26T02:23:42.615166+0000","last_unstale":"2024-04-26T02:23:43.660568+0000","last_undegraded":"2024-04-26T02:23:43.660568+0000","last_fullsized":"2024-04-26T02:23:43.660568+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:07:54.194247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.660750+0000","last_change":"2024-04-26T02:23:42.609572+0000","last_active":"2024-04-26T02:23:43.660750+0000","last_peered":"2024-04-26T02:23:43.660750+0000","last_clean":"2024-04-26T02:23:43.660750+0000","last_became_active":"2024-04-26T02:23:42.609019+0000","last_became_peered":"2024-04-26T02:23:42.609019+0000","last_unstale":"2024-04-26T02:23:43.660750+0000","last_undegraded":"2024-04-26T02:23:43.660750+0000","last_fullsized":"2024-04-26T02:23:43.660750+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:34:56.611120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.616952+0000","last_change":"2024-04-26T02:23:42.613541+0000","last_active":"2024-04-26T02:23:43.616952+0000","last_peered":"2024-04-26T02:23:43.616952+0000","last_clean":"2024-04-26T02:23:43.616952+0000","last_became_active":"2024-04-26T02:23:42.612975+0000","last_became_peered":"2024-04-26T02:23:42.612975+0000","last_unstale":"2024-04-26T02:23:43.616952+0000","last_undegraded":"2024-04-26T02:23:43.616952+0000","last_fullsized":"2024-04-26T02:23:43.616952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:52:20.904007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663364+0000","last_change":"2024-04-26T02:23:42.606688+0000","last_active":"2024-04-26T02:23:43.663364+0000","last_peered":"2024-04-26T02:23:43.663364+0000","last_clean":"2024-04-26T02:23:43.663364+0000","last_became_active":"2024-04-26T02:23:42.606239+0000","last_became_peered":"2024-04-26T02:23:42.606239+0000","last_unstale":"2024-04-26T02:23:43.663364+0000","last_undegraded":"2024-04-26T02:23:43.663364+0000","last_fullsized":"2024-04-26T02:23:43.663364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:16:06.174520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.660833+0000","last_change":"2024-04-26T02:23:42.620333+0000","last_active":"2024-04-26T02:23:43.660833+0000","last_peered":"2024-04-26T02:23:43.660833+0000","last_clean":"2024-04-26T02:23:43.660833+0000","last_became_active":"2024-04-26T02:23:42.619668+0000","last_became_peered":"2024-04-26T02:23:42.619668+0000","last_unstale":"2024-04-26T02:23:43.660833+0000","last_undegraded":"2024-04-26T02:23:43.660833+0000","last_fullsized":"2024-04-26T02:23:43.660833+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:47.928913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.618711+0000","last_change":"2024-04-26T02:23:42.623550+0000","last_active":"2024-04-26T02:23:43.618711+0000","last_peered":"2024-04-26T02:23:43.618711+0000","last_clean":"2024-04-26T02:23:43.618711+0000","last_became_active":"2024-04-26T02:23:42.622743+0000","last_became_peered":"2024-04-26T02:23:42.622743+0000","last_unstale":"2024-04-26T02:23:43.618711+0000","last_undegraded":"2024-04-26T02:23:43.618711+0000","last_fullsized":"2024-04-26T02:23:43.618711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:48.285504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.660826+0000","last_change":"2024-04-26T02:23:42.615565+0000","last_active":"2024-04-26T02:23:43.660826+0000","last_peered":"2024-04-26T02:23:43.660826+0000","last_clean":"2024-04-26T02:23:43.660826+0000","last_became_active":"2024-04-26T02:23:42.614991+0000","last_became_peered":"2024-04-26T02:23:42.614991+0000","last_unstale":"2024-04-26T02:23:43.660826+0000","last_undegraded":"2024-04-26T02:23:43.660826+0000","last_fullsized":"2024-04-26T02:23:43.660826+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:12:57.211761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663808+0000","last_change":"2024-04-26T02:23:42.607673+0000","last_active":"2024-04-26T02:23:43.663808+0000","last_peered":"2024-04-26T02:23:43.663808+0000","last_clean":"2024-04-26T02:23:43.663808+0000","last_became_active":"2024-04-26T02:23:42.607271+0000","last_became_peered":"2024-04-26T02:23:42.607271+0000","last_unstale":"2024-04-26T02:23:43.663808+0000","last_undegraded":"2024-04-26T02:23:43.663808+0000","last_fullsized":"2024-04-26T02:23:43.663808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:17:59.594516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663025+0000","last_change":"2024-04-26T02:23:42.617669+0000","last_active":"2024-04-26T02:23:43.663025+0000","last_peered":"2024-04-26T02:23:43.663025+0000","last_clean":"2024-04-26T02:23:43.663025+0000","last_became_active":"2024-04-26T02:23:42.617075+0000","last_became_peered":"2024-04-26T02:23:42.617075+0000","last_unstale":"2024-04-26T02:23:43.663025+0000","last_undegraded":"2024-04-26T02:23:43.663025+0000","last_fullsized":"2024-04-26T02:23:43.663025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:48.987055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663323+0000","last_change":"2024-04-26T02:23:42.607210+0000","last_active":"2024-04-26T02:23:43.663323+0000","last_peered":"2024-04-26T02:23:43.663323+0000","last_clean":"2024-04-26T02:23:43.663323+0000","last_became_active":"2024-04-26T02:23:42.606647+0000","last_became_peered":"2024-04-26T02:23:42.606647+0000","last_unstale":"2024-04-26T02:23:43.663323+0000","last_undegraded":"2024-04-26T02:23:43.663323+0000","last_fullsized":"2024-04-26T02:23:43.663323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:03:21.145208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663255+0000","last_change":"2024-04-26T02:23:42.615863+0000","last_active":"2024-04-26T02:23:43.663255+0000","last_peered":"2024-04-26T02:23:43.663255+0000","last_clean":"2024-04-26T02:23:43.663255+0000","last_became_active":"2024-04-26T02:23:42.614960+0000","last_became_peered":"2024-04-26T02:23:42.614960+0000","last_unstale":"2024-04-26T02:23:43.663255+0000","last_undegraded":"2024-04-26T02:23:43.663255+0000","last_fullsized":"2024-04-26T02:23:43.663255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:46:45.378184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-26T02:23:43.663130+0000","last_change":"2024-04-26T02:23:42.607875+0000","last_active":"2024-04-26T02:23:43.663130+0000","last_peered":"2024-04-26T02:23:43.663130+0000","last_clean":"2024-04-26T02:23:43.663130+0000","last_became_active":"2024-04-26T02:23:42.607094+0000","last_became_peered":"2024-04-26T02:23:42.607094+0000","last_unstale":"2024-04-26T02:23:43.663130+0000","last_undegraded":"2024-04-26T02:23:43.663130+0000","last_fullsized":"2024-04-26T02:23:43.663130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:50:35.730183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:23:49.681 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-04-26T02:23:49.681 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-04-26T02:23:49.682 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-04-26T02:23:49.726 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-04-26T02:23:49.726 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-04-26T02:23:49.808 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-04-26T02:23:49.808 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-04-26T02:23:49.893 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-04-26T02:23:49.894 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-04-26T02:23:49.982 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-04-26T02:23:49.982 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-04-26T02:23:50.066 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-04-26T02:23:50.066 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-04-26T02:23:50.146 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-04-26T02:23:50.146 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-04-26T02:23:50.232 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-04-26T02:23:50.232 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-04-26T02:23:50.271 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-04-26T02:23:50.272 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-04-26T02:23:50.353 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-04-26T02:23:50.354 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-04-26T02:23:50.434 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-04-26T02:23:50.434 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-04-26T02:23:50.520 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-04-26T02:23:50.520 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-04-26T02:23:50.601 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-04-26T02:23:50.601 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-04-26T02:23:50.680 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-04-26T02:23:50.680 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-04-26T02:23:50.766 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-04-26T02:23:50.766 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-04-26T02:23:50.848 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-04-26T02:23:50.848 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-04-26T02:23:50.934 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-04-26T02:23:50.935 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-04-26T02:23:51.018 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-04-26T02:23:51.019 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-04-26T02:23:51.102 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-04-26T02:23:51.103 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-04-26T02:23:51.181 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-04-26T02:23:51.181 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-04-26T02:23:51.262 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-04-26T02:23:51.262 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-04-26T02:23:51.347 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-04-26T02:23:51.347 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-04-26T02:23:51.431 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-04-26T02:23:51.431 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-04-26T02:23:51.520 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-04-26T02:23:51.520 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-04-26T02:23:51.599 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-04-26T02:23:51.599 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-04-26T02:23:51.681 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-04-26T02:23:51.682 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-04-26T02:23:51.766 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-04-26T02:23:51.767 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-04-26T02:23:51.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-04-26T02:23:52.030 INFO:teuthology.orchestra.run.smithi177.stdout:{} 2024-04-26T02:23:52.030 INFO:teuthology.orchestra.run.smithi177.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-04-26T02:23:52.045 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T02:23:52.045 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:23:52.118 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T02:23:52.118 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-04-26T02:23:52.401 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:23:52.400+0000 7fb190b23640 -1 mon.a@0(leader).osd e16 definitely_dead 0 2024-04-26T02:23:52.657 INFO:teuthology.orchestra.run.smithi177.stderr:marked down osd.0. 2024-04-26T02:23:52.684 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T02:23:52.684 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:23:52.777 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T02:23:52.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-04-26T02:23:53.072 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:23:53.072+0000 7fb190b23640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-04-26T02:23:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:marked down osd.3. 2024-04-26T02:23:53.683 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-04-26T02:23:53.684 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-04-26T02:23:54.697 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-04-26T02:23:54.697 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-04-26T02:23:54.741 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-04-26T02:23:54.741 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-04-26T02:23:54.830 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-04-26T02:23:54.831 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-04-26T02:23:54.912 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-04-26T02:23:54.912 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-04-26T02:23:55.002 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-04-26T02:23:55.003 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-04-26T02:23:55.042 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-04-26T02:23:55.042 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-04-26T02:23:55.122 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-04-26T02:23:55.122 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-04-26T02:23:55.206 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-04-26T02:23:55.206 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-04-26T02:23:55.293 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-04-26T02:23:55.293 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-04-26T02:23:55.333 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-04-26T02:23:55.334 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-04-26T02:23:55.418 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-04-26T02:23:55.419 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-04-26T02:23:55.500 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-04-26T02:23:55.500 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-04-26T02:23:55.589 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-04-26T02:23:55.589 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-04-26T02:23:55.628 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-04-26T02:23:55.628 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-04-26T02:23:55.718 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-04-26T02:23:55.719 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-04-26T02:23:55.799 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-04-26T02:23:55.799 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-04-26T02:23:55.884 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-04-26T02:23:55.884 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-04-26T02:23:55.969 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-04-26T02:23:55.970 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-04-26T02:23:56.052 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-04-26T02:23:56.052 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-04-26T02:23:56.142 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-04-26T02:23:56.142 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-04-26T02:23:56.181 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-04-26T02:23:56.181 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-04-26T02:23:56.262 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-04-26T02:23:56.262 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-04-26T02:23:56.342 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-04-26T02:23:56.342 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-04-26T02:23:56.423 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-04-26T02:23:56.423 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-04-26T02:23:56.512 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-04-26T02:23:56.512 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-04-26T02:23:56.551 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-04-26T02:23:56.551 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-04-26T02:23:56.635 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T02:23:56.635 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:24:01.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-26T02:24:02.736 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T02:24:02.737 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-04-26T02:24:03.051 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:24:03.048+0000 7fb190b23640 -1 mon.a@0(leader).osd e21 definitely_dead 0 2024-04-26T02:24:03.051 INFO:teuthology.orchestra.run.smithi177.stderr:osd.1 is already down. 2024-04-26T02:24:03.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-04-26T02:24:03.716 INFO:teuthology.orchestra.run.smithi177.stderr:marked osd lost in epoch 20 2024-04-26T02:24:03.737 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T02:24:03.737 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T02:24:03.739 INFO:tasks.ceph.osd.0:Started 2024-04-26T02:24:03.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T02:24:03.851 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-04-26T02:24:03.852 INFO:teuthology.orchestra.run.smithi177.stderr:0 2024-04-26T02:24:03.852 INFO:teuthology.orchestra.run.smithi177.stderr:1 2024-04-26T02:24:03.852 INFO:teuthology.orchestra.run.smithi177.stderr:2 2024-04-26T02:24:03.852 INFO:teuthology.orchestra.run.smithi177.stderr:abort 2024-04-26T02:24:03.852 INFO:teuthology.orchestra.run.smithi177.stderr:assert 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:config diff 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:config diff get 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:config get 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:config help [] 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:config set ... 2024-04-26T02:24:03.853 INFO:teuthology.orchestra.run.smithi177.stderr:admin_socket: invalid command 2024-04-26T02:24:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T02:24:03.855 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-26T02:24:04.341 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:24:04.340+0000 7f64643487c0 -1 Falling back to public interface 2024-04-26T02:24:07.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-26T02:24:08.090 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:24:08.088+0000 7f64643487c0 -1 osd.0 16 log_to_monitors true 2024-04-26T02:24:08.165 INFO:tasks.ceph.osd.0.smithi177.stderr:2024-04-26T02:24:08.164+0000 7f645c12e640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:24:08.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T02:24:08.987 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:08.987 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-04-26T02:24:08.987 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-04-26T02:24:08.987 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:09.001 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-04-26T02:24:09.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:24:09.309 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:09.309 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":24,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:24:09.159474+0000","last_up_change":"2024-04-26T02:24:09.159474+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-04-26T02:23:39.604824+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":2332209013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":2332209013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":2332209013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":2332209013}]},"public_addr":"172.21.15.177:6803/2332209013","cluster_addr":"172.21.15.177:6805/2332209013","heartbeat_back_addr":"172.21.15.177:6807/2332209013","heartbeat_front_addr":"172.21.15.177:6806/2332209013","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:24:09.330 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-04-26T02:24:09.330 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T02:24:09.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T02:24:09.333 INFO:tasks.ceph.osd.3:Started 2024-04-26T02:24:09.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T02:24:09.443 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-04-26T02:24:09.443 INFO:teuthology.orchestra.run.smithi177.stderr:0 2024-04-26T02:24:09.443 INFO:teuthology.orchestra.run.smithi177.stderr:1 2024-04-26T02:24:09.443 INFO:teuthology.orchestra.run.smithi177.stderr:2 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:abort 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:assert 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:config diff 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:config diff get 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:config get 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:config help [] 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:config set ... 2024-04-26T02:24:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:admin_socket: invalid command 2024-04-26T02:24:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T02:24:09.446 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-26T02:24:09.929 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:24:09.928+0000 7f9b80b887c0 -1 Falling back to public interface 2024-04-26T02:24:13.709 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:24:13.708+0000 7f9b80b887c0 -1 osd.3 17 log_to_monitors true 2024-04-26T02:24:14.446 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T02:24:14.472 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:24:14.472+0000 7fb193328640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-04-26T02:24:14.488 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-04-26T02:24:14.488+0000 7f9b74966640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:24:14.582 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-04-26T02:24:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-04-26T02:24:14.582 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:14.596 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-04-26T02:24:14.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:24:14.938 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:14.938 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":26,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:24:14.475139+0000","last_up_change":"2024-04-26T02:24:09.159474+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-04-26T02:23:39.604824+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":24,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":2332209013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":2332209013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":2332209013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":2332209013}]},"public_addr":"172.21.15.177:6803/2332209013","cluster_addr":"172.21.15.177:6805/2332209013","heartbeat_back_addr":"172.21.15.177:6807/2332209013","heartbeat_front_addr":"172.21.15.177:6806/2332209013","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3763693899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":3763693899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":3763693899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":3763693899}]},"public_addr":"172.21.15.177:6803/3763693899","cluster_addr":"172.21.15.177:6805/3763693899","heartbeat_back_addr":"172.21.15.177:6807/3763693899","heartbeat_front_addr":"172.21.15.177:6806/3763693899","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:24:17.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:24:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:18.260 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":28,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:24:16.486406+0000","last_up_change":"2024-04-26T02:24:15.483352+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-04-26T02:23:39.604824+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":27,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":2332209013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":2332209013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":2332209013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":2332209013}]},"public_addr":"172.21.15.177:6803/2332209013","cluster_addr":"172.21.15.177:6805/2332209013","heartbeat_back_addr":"172.21.15.177:6807/2332209013","heartbeat_front_addr":"172.21.15.177:6806/2332209013","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1599457430}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":1599457430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":1599457430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":1599457430}]},"public_addr":"172.21.15.177:6808/1599457430","cluster_addr":"172.21.15.177:6809/1599457430","heartbeat_back_addr":"172.21.15.177:6811/1599457430","heartbeat_front_addr":"172.21.15.177:6810/1599457430","state":["exists"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":27,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":252562504}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":252562504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":252562504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":252562504}]},"public_addr":"172.21.15.177:6808/252562504","cluster_addr":"172.21.15.177:6809/252562504","heartbeat_back_addr":"172.21.15.177:6811/252562504","heartbeat_front_addr":"172.21.15.177:6810/252562504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:24:18.279 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-04-26T02:24:18.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:24:18.280 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T02:24:18.280 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T02:24:18.433 INFO:teuthology.orchestra.run.smithi177.stdout:103079215108 2024-04-26T02:24:18.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:24:18.439 INFO:teuthology.orchestra.run.smithi177.stdout:38654705676 2024-04-26T02:24:18.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:24:18.441 INFO:teuthology.orchestra.run.smithi177.stdout:115964116994 2024-04-26T02:24:18.442 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:24:18.729 INFO:teuthology.orchestra.run.smithi177.stdout:103079215108 2024-04-26T02:24:18.746 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215108 got 103079215108 for osd.0 2024-04-26T02:24:18.746 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:18.758 INFO:teuthology.orchestra.run.smithi177.stdout:115964116994 2024-04-26T02:24:18.765 INFO:teuthology.orchestra.run.smithi177.stdout:38654705676 2024-04-26T02:24:18.776 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116994 got 115964116994 for osd.3 2024-04-26T02:24:18.777 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:18.782 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705676 got 38654705676 for osd.2 2024-04-26T02:24:18.782 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:18.782 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-04-26T02:24:18.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:19.082 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:19.097 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-04-26T02:24:18.422009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110124,"kb_used_data":1064,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377236,"statfs":{"total":386547056640,"available":386434289664,"internally_reserved":0,"allocated":1089536,"data_stored":513232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":111595817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002442"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-04-26T02:24:16.519671+0000","last_change":"2024-04-26T02:24:16.519505+0000","last_active":"2024-04-26T02:24:16.519671+0000","last_peered":"2024-04-26T02:24:16.519671+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:16.519671+0000","last_undegraded":"2024-04-26T02:24:16.519671+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":43,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.530028+0000","last_change":"2024-04-26T02:24:16.530882+0000","last_active":"2024-04-26T02:24:16.530028+0000","last_peered":"2024-04-26T02:24:16.530028+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:16.530028+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":55,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.531815+0000","last_change":"2024-04-26T02:24:16.532491+0000","last_active":"2024-04-26T02:24:16.531815+0000","last_peered":"2024-04-26T02:24:16.531815+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:16.531815+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":43,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.543416+0000","last_change":"2024-04-26T02:24:16.543986+0000","last_active":"2024-04-26T02:24:16.543416+0000","last_peered":"2024-04-26T02:24:16.543416+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:16.543416+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":31,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.549246+0000","last_change":"2024-04-26T02:24:16.549776+0000","last_active":"2024-04-26T02:24:16.549246+0000","last_peered":"2024-04-26T02:24:16.549246+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:16.549246+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-04-26T02:24:16.517650+0000","last_change":"2024-04-26T02:24:16.517484+0000","last_active":"2024-04-26T02:24:16.517650+0000","last_peered":"2024-04-26T02:24:16.517650+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:16.517650+0000","last_undegraded":"2024-04-26T02:24:16.517650+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":45,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.537911+0000","last_change":"2024-04-26T02:24:16.538519+0000","last_active":"2024-04-26T02:24:16.537911+0000","last_peered":"2024-04-26T02:24:16.537911+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:16.537911+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":44,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.540943+0000","last_change":"2024-04-26T02:24:16.541446+0000","last_active":"2024-04-26T02:24:16.540943+0000","last_peered":"2024-04-26T02:24:16.540943+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:16.540943+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":36,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.544468+0000","last_change":"2024-04-26T02:24:16.545078+0000","last_active":"2024-04-26T02:24:16.544468+0000","last_peered":"2024-04-26T02:24:16.544468+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:16.544468+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":53,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.536794+0000","last_change":"2024-04-26T02:24:16.537315+0000","last_active":"2024-04-26T02:24:16.536794+0000","last_peered":"2024-04-26T02:24:16.536794+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:16.536794+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":32,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.539132+0000","last_change":"2024-04-26T02:24:16.539774+0000","last_active":"2024-04-26T02:24:16.539132+0000","last_peered":"2024-04-26T02:24:16.539132+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:16.539132+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":17,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.548267+0000","last_change":"2024-04-26T02:24:16.548267+0000","last_active":"2024-04-26T02:24:16.548267+0000","last_peered":"2024-04-26T02:24:16.548267+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:16.548267+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":53,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.546004+0000","last_change":"2024-04-26T02:24:16.546515+0000","last_active":"2024-04-26T02:24:16.546004+0000","last_peered":"2024-04-26T02:24:16.546004+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:16.546004+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":31,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.535519+0000","last_change":"2024-04-26T02:24:16.535519+0000","last_active":"2024-04-26T02:24:16.535519+0000","last_peered":"2024-04-26T02:24:16.535519+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:16.535519+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":56,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.542009+0000","last_change":"2024-04-26T02:24:16.542692+0000","last_active":"2024-04-26T02:24:16.542009+0000","last_peered":"2024-04-26T02:24:16.542009+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:16.542009+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":44,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:16.533275+0000","last_change":"2024-04-26T02:24:16.533981+0000","last_active":"2024-04-26T02:24:16.533275+0000","last_peered":"2024-04-26T02:24:16.533275+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:16.533275+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964116994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":278528,"data_stored":119990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":28176080},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":193718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":180224,"data_stored":79534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":278528,"data_stored":119990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:19.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:24:19.421 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:19.421 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":55,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":3,"osd_up_since":1714098255,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":112766976,"bytes_avail":386434289664,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-26T02:23:23.479978+0000","services":{}},"progress_events":{}} 2024-04-26T02:24:19.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '13088462-cff2-40f2-960d-bd23a2ee33c5', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 55, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1714098255, 'num_in_osds': 4, 'osd_in_since': 1714098204, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 179368, 'bytes_used': 112766976, 'bytes_avail': 386434289664, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-04-26T02:23:23.479978+0000', 'services': {}}, 'progress_events': {}} 2024-04-26T02:24:19.438 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-04-26T02:24:19.438 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:24:19.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T02:24:19.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T02:24:19.590 INFO:teuthology.orchestra.run.smithi177.stdout:38654705677 2024-04-26T02:24:19.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:24:19.598 INFO:teuthology.orchestra.run.smithi177.stdout:115964116996 2024-04-26T02:24:19.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:24:19.603 INFO:teuthology.orchestra.run.smithi177.stdout:103079215109 2024-04-26T02:24:19.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:24:19.864 INFO:teuthology.orchestra.run.smithi177.stdout:38654705676 2024-04-26T02:24:19.882 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2024-04-26T02:24:19.928 INFO:teuthology.orchestra.run.smithi177.stdout:115964116994 2024-04-26T02:24:19.928 INFO:teuthology.orchestra.run.smithi177.stdout:103079215108 2024-04-26T02:24:19.947 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215109 got 103079215108 for osd.0 2024-04-26T02:24:19.947 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116996 got 115964116994 for osd.3 2024-04-26T02:24:20.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:24:20.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:24:20.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:24:21.188 INFO:teuthology.orchestra.run.smithi177.stdout:38654705677 2024-04-26T02:24:21.205 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705677 for osd.2 2024-04-26T02:24:21.205 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:21.227 INFO:teuthology.orchestra.run.smithi177.stdout:115964116996 2024-04-26T02:24:21.233 INFO:teuthology.orchestra.run.smithi177.stdout:103079215109 2024-04-26T02:24:21.246 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964116996 got 115964116996 for osd.3 2024-04-26T02:24:21.246 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:21.251 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215109 got 103079215109 for osd.0 2024-04-26T02:24:21.251 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:21.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:24:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:21.616 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":58,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":3,"osd_up_since":1714098255,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":112766976,"bytes_avail":386434289664,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-26T02:23:23.479978+0000","services":{}},"progress_events":{}} 2024-04-26T02:24:21.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '13088462-cff2-40f2-960d-bd23a2ee33c5', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 58, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 28, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1714098255, 'num_in_osds': 4, 'osd_in_since': 1714098204, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 179368, 'bytes_used': 112766976, 'bytes_avail': 386434289664, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-04-26T02:23:23.479978+0000', 'services': {}}, 'progress_events': {}} 2024-04-26T02:24:21.633 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-04-26T02:24:21.633 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-04-26T02:24:21.674 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stderr:INFO: op_size has been rounded to 8192 2024-04-26T02:24:21.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:hints = 1 2024-04-26T02:24:21.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-04-26T02:24:21.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Object prefix: benchmark_data_smithi177_33390 2024-04-26T02:24:21.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:24:21.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-04-26T02:24:22.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 1 20 1476 1456 11.3744 11.375 0.0177586 0.0136545 2024-04-26T02:24:23.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 2 20 2856 2836 11.0743 10.7812 0.010062 0.0140569 2024-04-26T02:24:24.675 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 3 20 4200 4180 10.8826 10.5 0.0166754 0.0143183 2024-04-26T02:24:25.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 4 20 5550 5530 10.7978 10.5469 0.0118091 0.0144516 2024-04-26T02:24:26.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 5 20 6830 6810 10.6381 10 0.0196626 0.0146605 2024-04-26T02:24:27.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 6 20 8083 8063 10.4965 9.78906 0.0151609 0.0148525 2024-04-26T02:24:28.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 7 20 9053 9033 10.0795 7.57812 0.0225097 0.0154776 2024-04-26T02:24:29.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 8 20 10225 10205 9.96404 9.15625 0.0372113 0.0156424 2024-04-26T02:24:30.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 9 20 11415 11395 9.88984 9.29688 0.0106268 0.0157818 2024-04-26T02:24:31.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:31.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 10 20 12291 12271 9.5852 6.84375 0.0181897 0.0162837 2024-04-26T02:24:32.449 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:32.449 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:32.504 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2024-04-26T02:24:32.426411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74169512,"num_objects":9060,"num_object_clones":0,"num_object_copies":36240,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":9114,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":9060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":72578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9087,"ondisk_log_size":9087,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":235016,"kb_used_data":101032,"kb_used_omap":11,"kb_used_meta":133940,"kb_avail":377252344,"statfs":{"total":386547056640,"available":386306400256,"internally_reserved":0,"allocated":103456768,"data_stored":102880464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":137154857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,7,11,6,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":73990144,"num_objects":9032,"num_object_clones":0,"num_object_copies":36128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003824"},"pg_stats":[{"pgid":"1.f","version":"28'635","reported_seq":662,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:29.506577+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:29.506577+0000","last_peered":"2024-04-26T02:24:29.506577+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:29.506577+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5201920,"num_objects":635,"num_object_clones":0,"num_object_copies":2540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":635,"num_write_kb":5080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'515","reported_seq":554,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.997429+0000","last_change":"2024-04-26T02:24:16.530882+0000","last_active":"2024-04-26T02:24:27.997429+0000","last_peered":"2024-04-26T02:24:27.997429+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:27.997429+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4198924,"num_objects":513,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":517,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":4116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":511},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'479","reported_seq":531,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.479446+0000","last_change":"2024-04-26T02:24:16.532491+0000","last_active":"2024-04-26T02:24:27.479446+0000","last_peered":"2024-04-26T02:24:27.479446+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:27.479446+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3905798,"num_objects":477,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":479,"num_write_kb":3822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":476},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'526","reported_seq":564,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:28.010293+0000","last_change":"2024-04-26T02:24:16.543986+0000","last_active":"2024-04-26T02:24:28.010293+0000","last_peered":"2024-04-26T02:24:28.010293+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:28.010293+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4289036,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":528,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":526,"num_write_kb":4197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":522},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'601","reported_seq":629,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:29.496361+0000","last_change":"2024-04-26T02:24:16.549776+0000","last_active":"2024-04-26T02:24:29.496361+0000","last_peered":"2024-04-26T02:24:29.496361+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:29.496361+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4911628,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":604,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":601,"num_write_kb":4805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":598},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'662","reported_seq":689,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:29.443750+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:29.443750+0000","last_peered":"2024-04-26T02:24:29.443750+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:29.443750+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5423104,"num_objects":662,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":662,"num_write_kb":5296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'542","reported_seq":581,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:28.011685+0000","last_change":"2024-04-26T02:24:16.538519+0000","last_active":"2024-04-26T02:24:28.011685+0000","last_peered":"2024-04-26T02:24:28.011685+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:28.011685+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4403724,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":542,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":4316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":536},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'559","reported_seq":598,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.962732+0000","last_change":"2024-04-26T02:24:16.541446+0000","last_active":"2024-04-26T02:24:27.962732+0000","last_peered":"2024-04-26T02:24:27.962732+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:27.962732+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4551180,"num_objects":556,"num_object_clones":0,"num_object_copies":2224,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":560,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":559,"num_write_kb":4460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":554},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'625","reported_seq":652,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:29.496697+0000","last_change":"2024-04-26T02:24:16.545078+0000","last_active":"2024-04-26T02:24:29.496697+0000","last_peered":"2024-04-26T02:24:29.496697+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:29.496697+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5071896,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":626,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":625,"num_write_kb":4977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":617},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'480","reported_seq":531,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.477386+0000","last_change":"2024-04-26T02:24:16.537315+0000","last_active":"2024-04-26T02:24:27.477386+0000","last_peered":"2024-04-26T02:24:27.477386+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:27.477386+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3928588,"num_objects":480,"num_object_clones":0,"num_object_copies":1920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":3838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":478},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'644","reported_seq":671,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:29.494333+0000","last_change":"2024-04-26T02:24:16.539774+0000","last_active":"2024-04-26T02:24:29.494333+0000","last_peered":"2024-04-26T02:24:29.494333+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:29.494333+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5253906,"num_objects":642,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":648,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":5140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":639},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'621","reported_seq":637,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:29.496455+0000","last_change":"2024-04-26T02:24:16.548267+0000","last_active":"2024-04-26T02:24:29.496455+0000","last_peered":"2024-04-26T02:24:29.496455+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:29.496455+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5085446,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":623,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":621,"num_write_kb":4967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":620},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'511","reported_seq":563,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.497305+0000","last_change":"2024-04-26T02:24:16.546515+0000","last_active":"2024-04-26T02:24:27.497305+0000","last_peered":"2024-04-26T02:24:27.497305+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:27.497305+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4184326,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":513,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":4087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":510},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'622","reported_seq":649,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:29.453309+0000","last_change":"2024-04-26T02:24:16.535519+0000","last_active":"2024-04-26T02:24:29.453309+0000","last_peered":"2024-04-26T02:24:29.453309+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:29.453309+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075468,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":624,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":622,"num_write_kb":4972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":618},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'513","reported_seq":566,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:27.486992+0000","last_change":"2024-04-26T02:24:16.542692+0000","last_active":"2024-04-26T02:24:27.486992+0000","last_peered":"2024-04-26T02:24:27.486992+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:27.486992+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4190732,"num_objects":512,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":516,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":513,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":510},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'552","reported_seq":590,"reported_epoch":28,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-04-26T02:24:28.025749+0000","last_change":"2024-04-26T02:24:16.533981+0000","last_active":"2024-04-26T02:24:28.025749+0000","last_peered":"2024-04-26T02:24:28.025749+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:28.025749+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4493836,"num_objects":549,"num_object_clones":0,"num_object_copies":2196,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":553,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":552,"num_write_kb":4404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":547},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74169512,"num_objects":9060,"num_object_clones":0,"num_object_copies":36240,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":9114,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":9060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":72578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108666880,"data_stored":108666880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9087,"ondisk_log_size":9087,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964116998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65276,"kb_used_data":37748,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94306564,"statfs":{"total":96636764160,"available":96569921536,"internally_reserved":0,"allocated":38653952,"data_stored":38495414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":28176080},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64596,"kb_used_data":29260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307244,"statfs":{"total":96636764160,"available":96570617856,"internally_reserved":0,"allocated":29962240,"data_stored":29803702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":180224,"data_stored":79534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":33848,"kb_used_omap":4,"kb_used_meta":44091,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":34660352,"data_stored":34501814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":45149903},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35053568,"data_stored":35053568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32362496,"data_stored":32362496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41209856,"data_stored":41209856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:32.505 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-04-26T02:24:32.505 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:32.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-04-26T02:24:32.676 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 11 20 13299 13279 9.42968 7.875 0.012863 0.0165437 2024-04-26T02:24:32.824 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:32.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_4", 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1224592654, 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'1", 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_6", 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3101443614, 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:32.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'2", 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:32.828 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:32.829 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:32.829 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:32.829 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:32.829 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:32.829 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:32.886 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:32.886 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-04-26T02:24:32.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-04-26T02:24:33.012 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:33.028 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:33.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-04-26T02:24:33.264 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 1, 2024-04-26T02:24:33.265 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 1, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_7", 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 2358579165, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:33.266 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "2(0)" 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(1)", 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:33.267 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:33.268 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:33.329 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-04-26T02:24:33.329 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-04-26T02:24:33.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-04-26T02:24:33.584 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-26T02:24:33.611 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:33.611 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-04-26T02:24:33.677 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 12 20 13932 13912 9.05597 4.94531 0.0137346 0.0172324 2024-04-26T02:24:33.783 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_5", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 4248945696, 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_5", 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 4212789968, 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:33.788 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:33.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:33.788 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:33.788 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:33.788 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:33.797 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:33.797 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-04-26T02:24:33.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-04-26T02:24:34.316 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:34.374 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:34.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-04-26T02:24:34.669 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:24:34.667+0000 7fb193328640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 13 pgs recovery_unfound (PG_DAMAGED) 2024-04-26T02:24:34.677 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 13 20 14343 14323 8.60607 3.21094 0.0300462 0.0181377 2024-04-26T02:24:34.756 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_3", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 459768081, 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'1", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:34.758 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_3", 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 480221105, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:34.759 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'2", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(1)", 2024-04-26T02:24:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:34.761 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:34.761 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:34.761 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:34.761 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:34.817 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-04-26T02:24:34.817 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-04-26T02:24:34.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-04-26T02:24:35.134 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:35.170 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-04-26T02:24:35.170 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:35.170 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-04-26T02:24:35.389 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.390 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_6", 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1708673474, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'5", 2024-04-26T02:24:35.391 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_9", 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 831402034, 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:35.393 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.393 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'6", 2024-04-26T02:24:35.393 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:35.394 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(2)", 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:35.395 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:35.403 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-04-26T02:24:35.403 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-04-26T02:24:35.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-04-26T02:24:35.596 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:35.642 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:35.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-04-26T02:24:35.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 14 20 15152 15132 8.43929 6.32031 0.028034 0.0184826 2024-04-26T02:24:35.955 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:35.957 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:35.957 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:35.957 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_8", 2024-04-26T02:24:35.958 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1901787523, 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'5", 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'3", 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_1", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3122379363, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:35.961 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:35.962 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:35.962 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:35.962 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:36.027 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:36.027 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-04-26T02:24:36.027 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-04-26T02:24:36.197 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:36.213 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:36.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-04-26T02:24:36.474 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 3, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 3, 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:36.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_4", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 873891716, 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'8", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:36.476 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_7", 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3996155460, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'9", 2024-04-26T02:24:36.477 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'6", 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_2", 2024-04-26T02:24:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1767667508, 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'7", 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:36.479 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(3)", 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:36.480 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:36.481 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:36.525 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-04-26T02:24:36.525 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-04-26T02:24:36.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-04-26T02:24:36.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 15 20 15972 15952 8.30378 6.40625 0.0120239 0.0187938 2024-04-26T02:24:36.888 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 3 objects unfound and apparently lost marking 2024-04-26T02:24:36.929 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:36.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:37.297 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_1", 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:37.298 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 26281365, 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'1", 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: "2(0)" 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:37.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_2", 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3196340981, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'2", 2024-04-26T02:24:37.300 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "2(0)" 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(3)", 2024-04-26T02:24:37.301 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:37.302 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:37.302 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:37.302 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:37.302 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:37.326 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-04-26T02:24:37.326 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-04-26T02:24:37.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-04-26T02:24:37.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 16 20 16919 16899 8.2469 7.39844 0.0281086 0.018922 2024-04-26T02:24:37.699 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:37.747 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:37.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.154 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 3, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 3, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_8", 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1861424966, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'5", 2024-04-26T02:24:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_6", 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3910131350, 2024-04-26T02:24:38.156 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_4", 2024-04-26T02:24:38.157 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 2787839990, 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: "2(3)" 2024-04-26T02:24:38.158 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(2)", 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:38.159 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:38.218 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-04-26T02:24:38.218 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-04-26T02:24:38.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-04-26T02:24:38.680 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 3 objects unfound and apparently lost marking 2024-04-26T02:24:38.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 17 20 17876 17856 8.20157 7.47656 0.0271037 0.0190358 2024-04-26T02:24:38.707 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:38.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-04-26T02:24:38.864 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 1, 2024-04-26T02:24:38.865 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 1, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_7", 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 3761017527, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'1", 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:38.866 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:38.867 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:38.868 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:38.868 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:38.878 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:38.879 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-04-26T02:24:38.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-04-26T02:24:39.107 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-26T02:24:39.153 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:39.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 1, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 1, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_8", 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 2730327960, 2024-04-26T02:24:39.447 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'1", 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "2(1)" 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:39.448 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:39.449 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:39.485 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:39.485 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-04-26T02:24:39.486 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-04-26T02:24:39.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 18 20 18490 18470 8.01248 4.79688 0.0107656 0.0194344 2024-04-26T02:24:39.866 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 1 objects unfound and apparently lost marking 2024-04-26T02:24:39.939 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:39.939 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:40.195 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_5", 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 2998549577, 2024-04-26T02:24:40.196 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'2", 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_2", 2024-04-26T02:24:40.197 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 4032093161, 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'1", 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:40.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:40.199 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:40.270 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:40.270 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-04-26T02:24:40.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-04-26T02:24:40.666 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:40.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 19 20 19245 19225 7.90123 5.89844 0.0175404 0.0197608 2024-04-26T02:24:40.714 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:40.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-04-26T02:24:40.718 INFO:tasks.ceph.mon.a.smithi177.stderr:2024-04-26T02:24:40.715+0000 7fb193328640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_9", 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 1217469130, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'3", 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'1", 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:40.879 INFO:teuthology.orchestra.run.smithi177.stdout: "2(1)" 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "new_9", 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 170713514, 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:40.882 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'2", 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "2(1)" 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:40.883 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:40.884 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:40.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:40.884 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:40.884 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:40.884 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:40.890 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:40.891 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-04-26T02:24:40.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-04-26T02:24:41.182 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:41.252 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-04-26T02:24:41.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-04-26T02:24:41.608 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "offset": { 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "", 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": 0, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 0, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": -9223372036854775808, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "num_missing": 2, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "num_unfound": 2, 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "objects": [ 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:41.609 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existed_3", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 943960171, 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'5", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "0'0", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-26T02:24:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": { 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "oid": "existing_1", 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "key": "", 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": -2, 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "hash": 2885884443, 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 0, 2024-04-26T02:24:41.611 INFO:teuthology.orchestra.run.smithi177.stdout: "pool": 1, 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "namespace": "" 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "need": "19'4", 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "have": "16'1", 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "flags": "none", 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "clean_regions": "clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false", 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "locations": [ 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "2(2)" 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "state": "NotRecovering", 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "available_might_have_unfound": true, 2024-04-26T02:24:41.612 INFO:teuthology.orchestra.run.smithi177.stdout: "might_have_unfound": [ 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": "1(0)", 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: "status": "osd is down" 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout: "more": false 2024-04-26T02:24:41.613 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:41.649 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6406, 18446744073709545209)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-04-26T02:24:41.649 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-04-26T02:24:41.649 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-04-26T02:24:41.684 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:24:41.687568+0000 min lat: 0.00281663 max lat: 0.637369 avg lat: 0.0200109 2024-04-26T02:24:41.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:24:41.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 20 20 20005 19985 7.80306 5.9375 0.0239881 0.0200109 2024-04-26T02:24:42.008 INFO:teuthology.orchestra.run.smithi177.stderr:pg has 2 objects unfound and apparently lost marking 2024-04-26T02:24:42.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-04-26T02:24:42.233 INFO:teuthology.orchestra.run.smithi177.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-04-26T02:24:42.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-04-26T02:24:42.521 INFO:teuthology.orchestra.run.smithi177.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-04-26T02:24:42.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-04-26T02:24:42.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 21 20 20742 20722 7.70559 5.75781 0.0303804 0.0202603 2024-04-26T02:24:42.826 INFO:teuthology.orchestra.run.smithi177.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-04-26T02:24:42.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:24:42.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T02:24:42.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T02:24:43.393 INFO:teuthology.orchestra.run.smithi177.stdout:38654705683 2024-04-26T02:24:43.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:24:43.409 INFO:teuthology.orchestra.run.smithi177.stdout:115964117001 2024-04-26T02:24:43.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:24:43.538 INFO:teuthology.orchestra.run.smithi177.stdout:103079215115 2024-04-26T02:24:43.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:24:43.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 22 20 21594 21574 7.65788 6.65625 0.0196079 0.0203807 2024-04-26T02:24:44.285 INFO:teuthology.orchestra.run.smithi177.stdout:38654705681 2024-04-26T02:24:44.329 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705683 got 38654705681 for osd.2 2024-04-26T02:24:44.540 INFO:teuthology.orchestra.run.smithi177.stdout:115964117000 2024-04-26T02:24:44.607 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964117001 got 115964117000 for osd.3 2024-04-26T02:24:44.628 INFO:teuthology.orchestra.run.smithi177.stdout:103079215113 2024-04-26T02:24:44.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 23 20 22436 22416 7.61093 6.57812 0.0278441 0.0205181 2024-04-26T02:24:44.687 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215115 got 103079215113 for osd.0 2024-04-26T02:24:45.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:24:45.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:24:45.685 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 24 20 23359 23339 7.59423 7.21094 0.00902755 0.0205601 2024-04-26T02:24:45.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:24:46.319 INFO:teuthology.orchestra.run.smithi177.stdout:38654705683 2024-04-26T02:24:46.354 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705683 got 38654705683 for osd.2 2024-04-26T02:24:46.355 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:46.587 INFO:teuthology.orchestra.run.smithi177.stdout:115964117001 2024-04-26T02:24:46.624 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 115964117001 got 115964117001 for osd.3 2024-04-26T02:24:46.624 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:46.686 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 25 20 24178 24158 7.54613 6.39844 0.0279419 0.020691 2024-04-26T02:24:46.894 INFO:teuthology.orchestra.run.smithi177.stdout:103079215115 2024-04-26T02:24:46.911 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215115 got 103079215115 for osd.0 2024-04-26T02:24:46.912 DEBUG:teuthology.parallel:result is None 2024-04-26T02:24:46.912 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-04-26T02:24:46.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:47.450 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:47.451 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:47.489 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-04-26T02:24:46.430847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":178535592,"num_objects":21800,"num_object_clones":0,"num_object_copies":87200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21827,"num_write_kb":174498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21854,"ondisk_log_size":21854,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":494216,"kb_used_data":256104,"kb_used_omap":11,"kb_used_meta":238068,"kb_avail":376993144,"statfs":{"total":386547056640,"available":386040979456,"internally_reserved":0,"allocated":262250496,"data_stored":261674192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":243781929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,10,11,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":80568320,"num_objects":9835,"num_object_clones":0,"num_object_copies":39340,"num_objects_missing_on_primary":-19,"num_objects_missing":-19,"num_objects_degraded":9785,"num_objects_misplaced":0,"num_objects_unfound":-25,"num_objects_dirty":9835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":78680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003847"},"pg_stats":[{"pgid":"1.f","version":"28'1342","reported_seq":1367,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.543405+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:44.543405+0000","last_peered":"2024-04-26T02:24:44.543405+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:44.543405+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993664,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":10736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1349","reported_seq":1398,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.390721+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:43.390721+0000","last_peered":"2024-04-26T02:24:43.390721+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:43.390721+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.070758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11014668,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1347,"num_write_kb":10772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1296","reported_seq":1355,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.327493+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:43.327493+0000","last_peered":"2024-04-26T02:24:43.327493+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:43.327493+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.005601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10590470,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1321","reported_seq":1370,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.435005+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:43.435005+0000","last_peered":"2024-04-26T02:24:43.435005+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:43.435005+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1321,"log_dups_size":0,"ondisk_log_size":1321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:45.084522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10785292,"num_objects":1317,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":10541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1389","reported_seq":1426,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.462336+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:44.462336+0000","last_peered":"2024-04-26T02:24:44.462336+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:44.462336+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.904495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11350540,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":11093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1410","reported_seq":1437,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.513540+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:44.513540+0000","last_peered":"2024-04-26T02:24:44.513540+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:44.513540+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11550720,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1410,"num_write_kb":11280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1410}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'1364","reported_seq":1413,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.395729+0000","last_change":"2024-04-26T02:24:35.636091+0000","last_active":"2024-04-26T02:24:43.395729+0000","last_peered":"2024-04-26T02:24:43.395729+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:43.395729+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.114813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11121164,"num_objects":1358,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1358}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1407","reported_seq":1456,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.419523+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:43.419523+0000","last_peered":"2024-04-26T02:24:43.419523+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:43.419523+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.094375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11481612,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1405,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1393","reported_seq":1433,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.472170+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:44.472170+0000","last_peered":"2024-04-26T02:24:44.472170+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:44.472170+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1393,"log_dups_size":0,"ondisk_log_size":1393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.942417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11338776,"num_objects":1385,"num_object_clones":0,"num_object_copies":5540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1390,"num_write_kb":11097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1385}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1317","reported_seq":1378,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.266342+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:43.266342+0000","last_peered":"2024-04-26T02:24:43.266342+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:43.266342+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10768908,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1315,"num_write_kb":10518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1315}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1383","reported_seq":1423,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.513020+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:44.513020+0000","last_peered":"2024-04-26T02:24:44.513020+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:44.513020+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.921341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11283218,"num_objects":1378,"num_object_clones":0,"num_object_copies":5512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1380,"num_write_kb":11028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1366","reported_seq":1389,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.484311+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:44.484311+0000","last_peered":"2024-04-26T02:24:44.484311+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:44.484311+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.921363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11180294,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":10919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1355","reported_seq":1414,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.327021+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:43.327021+0000","last_peered":"2024-04-26T02:24:43.327021+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:43.327021+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.968478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11090182,"num_objects":1354,"num_object_clones":0,"num_object_copies":5416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":10831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1354}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1418","reported_seq":1452,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.492001+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:44.492001+0000","last_peered":"2024-04-26T02:24:44.492001+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:44.492001+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:45.877858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11579916,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1416,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1375","reported_seq":1438,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.344589+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:43.344589+0000","last_peered":"2024-04-26T02:24:43.344589+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:43.344589+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1375,"log_dups_size":0,"ondisk_log_size":1375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.883711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11235852,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1373,"num_write_kb":10981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1369","reported_seq":1418,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.367206+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:43.367206+0000","last_peered":"2024-04-26T02:24:43.367206+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:43.367206+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.094385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11170316,"num_objects":1364,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":10924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":178535592,"num_objects":21800,"num_object_clones":0,"num_object_copies":87200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21827,"num_write_kb":174498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266264576,"data_stored":266264576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21854,"ondisk_log_size":21854,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155320,"kb_used_data":87536,"kb_used_omap":4,"kb_used_meta":67771,"kb_avail":94216520,"statfs":{"total":96636764160,"available":96477716480,"internally_reserved":0,"allocated":89636864,"data_stored":89478326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":69398224},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163612,"kb_used_data":83860,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94208228,"statfs":{"total":96636764160,"available":96469225472,"internally_reserved":0,"allocated":85872640,"data_stored":85714102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,7,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":180224,"data_stored":79534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148092,"kb_used_data":84532,"kb_used_omap":4,"kb_used_meta":63547,"kb_avail":94223748,"statfs":{"total":96636764160,"available":96485117952,"internally_reserved":0,"allocated":86560768,"data_stored":86402230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":65072847},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87584768,"data_stored":87584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87306240,"data_stored":87306240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91332608,"data_stored":91332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:47.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:47.686 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 26 20 24912 24892 7.47645 5.73438 0.019776 0.0208832 2024-04-26T02:24:48.400 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:48.401 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:48.438 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-04-26T02:24:46.430847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":178535592,"num_objects":21800,"num_object_clones":0,"num_object_copies":87200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21827,"num_write_kb":174498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21854,"ondisk_log_size":21854,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":494216,"kb_used_data":256104,"kb_used_omap":11,"kb_used_meta":238068,"kb_avail":376993144,"statfs":{"total":386547056640,"available":386040979456,"internally_reserved":0,"allocated":262250496,"data_stored":261674192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11991,"internal_metadata":243781929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,10,11,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":80568320,"num_objects":9835,"num_object_clones":0,"num_object_copies":39340,"num_objects_missing_on_primary":-19,"num_objects_missing":-19,"num_objects_degraded":9785,"num_objects_misplaced":0,"num_objects_unfound":-25,"num_objects_dirty":9835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":78680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003847"},"pg_stats":[{"pgid":"1.f","version":"28'1342","reported_seq":1367,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.543405+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:44.543405+0000","last_peered":"2024-04-26T02:24:44.543405+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:44.543405+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993664,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":10736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1349","reported_seq":1398,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.390721+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:43.390721+0000","last_peered":"2024-04-26T02:24:43.390721+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:43.390721+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.070758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11014668,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1347,"num_write_kb":10772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1296","reported_seq":1355,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.327493+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:43.327493+0000","last_peered":"2024-04-26T02:24:43.327493+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:43.327493+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.005601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10590470,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":10350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1321","reported_seq":1370,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.435005+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:43.435005+0000","last_peered":"2024-04-26T02:24:43.435005+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:43.435005+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1321,"log_dups_size":0,"ondisk_log_size":1321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:45.084522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10785292,"num_objects":1317,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":10541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1389","reported_seq":1426,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.462336+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:44.462336+0000","last_peered":"2024-04-26T02:24:44.462336+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:44.462336+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.904495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11350540,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":11093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1410","reported_seq":1437,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.513540+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:44.513540+0000","last_peered":"2024-04-26T02:24:44.513540+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:44.513540+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11550720,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1410,"num_write_kb":11280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1410}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'1364","reported_seq":1413,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.395729+0000","last_change":"2024-04-26T02:24:35.636091+0000","last_active":"2024-04-26T02:24:43.395729+0000","last_peered":"2024-04-26T02:24:43.395729+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:43.395729+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.114813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11121164,"num_objects":1358,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1362,"num_write_kb":10876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1358}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1407","reported_seq":1456,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.419523+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:43.419523+0000","last_peered":"2024-04-26T02:24:43.419523+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:43.419523+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.094375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11481612,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1405,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1393","reported_seq":1433,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.472170+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:44.472170+0000","last_peered":"2024-04-26T02:24:44.472170+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:44.472170+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1393,"log_dups_size":0,"ondisk_log_size":1393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.942417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11338776,"num_objects":1385,"num_object_clones":0,"num_object_copies":5540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1390,"num_write_kb":11097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1385}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1317","reported_seq":1378,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.266342+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:43.266342+0000","last_peered":"2024-04-26T02:24:43.266342+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:43.266342+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10768908,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1315,"num_write_kb":10518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1315}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1383","reported_seq":1423,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.513020+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:44.513020+0000","last_peered":"2024-04-26T02:24:44.513020+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:44.513020+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.921341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11283218,"num_objects":1378,"num_object_clones":0,"num_object_copies":5512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1380,"num_write_kb":11028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1366","reported_seq":1389,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.484311+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:44.484311+0000","last_peered":"2024-04-26T02:24:44.484311+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:44.484311+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.921363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11180294,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":10919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1355","reported_seq":1414,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.327021+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:43.327021+0000","last_peered":"2024-04-26T02:24:43.327021+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:43.327021+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:44.968478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11090182,"num_objects":1354,"num_object_clones":0,"num_object_copies":5416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":10831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1354}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1418","reported_seq":1452,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:44.492001+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:44.492001+0000","last_peered":"2024-04-26T02:24:44.492001+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:44.492001+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:45.877858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11579916,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1416,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1375","reported_seq":1438,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.344589+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:43.344589+0000","last_peered":"2024-04-26T02:24:43.344589+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:43.344589+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1375,"log_dups_size":0,"ondisk_log_size":1375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:46.883711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11235852,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1373,"num_write_kb":10981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1369","reported_seq":1418,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:43.367206+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:43.367206+0000","last_peered":"2024-04-26T02:24:43.367206+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:43.367206+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:47.094385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11170316,"num_objects":1364,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":10924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":178535592,"num_objects":21800,"num_object_clones":0,"num_object_copies":87200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21827,"num_write_kb":174498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266264576,"data_stored":266264576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21854,"ondisk_log_size":21854,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155320,"kb_used_data":87536,"kb_used_omap":4,"kb_used_meta":67771,"kb_avail":94216520,"statfs":{"total":96636764160,"available":96477716480,"internally_reserved":0,"allocated":89636864,"data_stored":89478326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":69398224},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163612,"kb_used_data":83860,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94208228,"statfs":{"total":96636764160,"available":96469225472,"internally_reserved":0,"allocated":85872640,"data_stored":85714102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,7,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":180224,"data_stored":79534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148092,"kb_used_data":84532,"kb_used_omap":4,"kb_used_meta":63547,"kb_avail":94223748,"statfs":{"total":96636764160,"available":96485117952,"internally_reserved":0,"allocated":86560768,"data_stored":86402230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":65072847},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87584768,"data_stored":87584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87306240,"data_stored":87306240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91332608,"data_stored":91332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:48.441 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-04-26T02:24:48.441 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-04-26T02:24:48.442 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-04-26T02:24:48.571 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-04-26T02:24:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:48.584 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-04-26T02:24:48.584 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-04-26T02:24:48.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 27 20 25749 25729 7.44161 6.53906 0.0173701 0.0209803 2024-04-26T02:24:48.702 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-04-26T02:24:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:48.722 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-04-26T02:24:48.722 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-04-26T02:24:48.815 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-04-26T02:24:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:48.825 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-04-26T02:24:48.825 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-04-26T02:24:48.924 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-04-26T02:24:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:48.927 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-04-26T02:24:48.927 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-04-26T02:24:49.018 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-04-26T02:24:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.022 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-04-26T02:24:49.023 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-04-26T02:24:49.127 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-04-26T02:24:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.130 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-04-26T02:24:49.131 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-04-26T02:24:49.255 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-04-26T02:24:49.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.263 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-04-26T02:24:49.263 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-04-26T02:24:49.332 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-04-26T02:24:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.336 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-04-26T02:24:49.336 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-04-26T02:24:49.456 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-04-26T02:24:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.464 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-04-26T02:24:49.465 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-04-26T02:24:49.607 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-04-26T02:24:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.613 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-04-26T02:24:49.614 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-04-26T02:24:49.656 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-04-26T02:24:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.659 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-04-26T02:24:49.659 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-04-26T02:24:49.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 28 20 26455 26435 7.37282 5.51562 0.0028054 0.0211343 2024-04-26T02:24:49.717 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-04-26T02:24:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.720 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-04-26T02:24:49.720 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-04-26T02:24:49.800 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-04-26T02:24:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.803 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-04-26T02:24:49.803 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-04-26T02:24:49.954 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-04-26T02:24:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:49.957 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-04-26T02:24:49.957 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-04-26T02:24:50.034 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-04-26T02:24:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.045 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-04-26T02:24:50.045 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-04-26T02:24:50.139 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-04-26T02:24:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.144 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-04-26T02:24:50.144 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-04-26T02:24:50.269 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-04-26T02:24:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.275 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-04-26T02:24:50.275 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-04-26T02:24:50.367 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-04-26T02:24:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.376 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-04-26T02:24:50.376 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-04-26T02:24:50.485 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-04-26T02:24:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.495 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-04-26T02:24:50.495 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-04-26T02:24:50.636 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-04-26T02:24:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.646 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-04-26T02:24:50.646 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-04-26T02:24:50.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 29 20 27171 27151 7.31148 5.59375 0.0264525 0.0213562 2024-04-26T02:24:50.775 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-04-26T02:24:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.790 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-04-26T02:24:50.790 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-04-26T02:24:50.946 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-04-26T02:24:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:50.957 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-04-26T02:24:50.957 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-04-26T02:24:51.046 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-04-26T02:24:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:51.056 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-04-26T02:24:51.057 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-04-26T02:24:51.126 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-04-26T02:24:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:51.131 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-04-26T02:24:51.132 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-04-26T02:24:51.217 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-04-26T02:24:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:51.227 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-04-26T02:24:51.228 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-04-26T02:24:51.372 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-04-26T02:24:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:51.398 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-04-26T02:24:51.398 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-04-26T02:24:51.476 INFO:teuthology.orchestra.run.smithi177.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-04-26T02:24:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:24:51.480 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T02:24:51.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T02:24:51.482 INFO:tasks.ceph.osd.1:Started 2024-04-26T02:24:51.483 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T02:24:51.687 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 30 20 27983 27963 7.27921 6.34375 0.0196101 0.0214518 2024-04-26T02:24:51.807 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-04-26T02:24:51.807 INFO:teuthology.orchestra.run.smithi177.stderr:0 2024-04-26T02:24:51.807 INFO:teuthology.orchestra.run.smithi177.stderr:1 2024-04-26T02:24:51.807 INFO:teuthology.orchestra.run.smithi177.stderr:2 2024-04-26T02:24:51.807 INFO:teuthology.orchestra.run.smithi177.stderr:abort 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:assert 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:config diff 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:config diff get 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:config get 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:config help [] 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:config set ... 2024-04-26T02:24:51.808 INFO:teuthology.orchestra.run.smithi177.stderr:admin_socket: invalid command 2024-04-26T02:24:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T02:24:51.811 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-26T02:24:52.173 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:24:52.174+0000 7f3c0b1567c0 -1 Falling back to public interface 2024-04-26T02:24:52.688 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 31 20 28840 28820 7.26019 6.69531 0.01736 0.0215086 2024-04-26T02:24:53.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 32 20 29701 29681 7.24334 6.72656 0.0210062 0.0215588 2024-04-26T02:24:54.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 33 20 30513 30493 7.21607 6.34375 0.0111738 0.0216439 2024-04-26T02:24:55.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 34 20 31293 31273 7.18306 6.09375 0.0300034 0.0217399 2024-04-26T02:24:56.190 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:24:56.190+0000 7f3c0b1567c0 -1 osd.1 19 log_to_monitors true 2024-04-26T02:24:56.537 INFO:tasks.ceph.osd.1.smithi177.stderr:2024-04-26T02:24:56.538+0000 7f3bfef34640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:24:56.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 35 20 32067 32047 7.15059 6.04688 0.0301356 0.0218402 2024-04-26T02:24:56.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T02:24:57.198 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:24:57.198 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-04-26T02:24:57.199 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-04-26T02:24:57.199 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:24:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-04-26T02:24:57.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:24:57.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 36 20 32720 32700 7.09367 5.10156 0.0335152 0.0219152 2024-04-26T02:24:57.692 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:57.692 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":30,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:24:57.516281+0000","last_up_change":"2024-04-26T02:24:57.516281+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-04-26T02:23:39.604824+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":27,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":2332209013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":2332209013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":2332209013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":2332209013}]},"public_addr":"172.21.15.177:6803/2332209013","cluster_addr":"172.21.15.177:6805/2332209013","heartbeat_back_addr":"172.21.15.177:6807/2332209013","heartbeat_front_addr":"172.21.15.177:6806/2332209013","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":30,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3273495033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":3273495033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":3273495033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":3273495033}]},"public_addr":"172.21.15.177:6812/3273495033","cluster_addr":"172.21.15.177:6813/3273495033","heartbeat_back_addr":"172.21.15.177:6815/3273495033","heartbeat_front_addr":"172.21.15.177:6814/3273495033","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":27,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":252562504}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":252562504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":252562504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":252562504}]},"public_addr":"172.21.15.177:6808/252562504","cluster_addr":"172.21.15.177:6809/252562504","heartbeat_back_addr":"172.21.15.177:6811/252562504","heartbeat_front_addr":"172.21.15.177:6810/252562504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:24:57.708 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-04-26T02:24:57.708 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-04-26T02:24:57.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:57.976 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:57.976 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:57.991 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-04-26T02:24:57.526054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242613416,"num_objects":29622,"num_object_clones":0,"num_object_copies":118488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29649,"num_write_kb":237074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29676,"ondisk_log_size":29676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":701024,"kb_used_data":350016,"kb_used_omap":14,"kb_used_meta":350961,"kb_avail":376786336,"statfs":{"total":386547056640,"available":385829208064,"internally_reserved":0,"allocated":358416384,"data_stored":357825449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":359384627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,7,12,16,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46546944,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.094757"},"pg_stats":[{"pgid":"1.f","version":"28'1848","reported_seq":1875,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.484534+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:54.484534+0000","last_peered":"2024-04-26T02:24:54.484534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:54.484534+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1848,"log_dups_size":0,"ondisk_log_size":1848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1846","reported_seq":1895,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.163940+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:53.163940+0000","last_peered":"2024-04-26T02:24:53.163940+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:53.163940+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086092,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":14748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1782","reported_seq":1841,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:52.532694+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:52.532694+0000","last_peered":"2024-04-26T02:24:52.532694+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:52.532694+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:54.991136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571782,"num_objects":1779,"num_object_clones":0,"num_object_copies":7116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1781,"num_write_kb":14238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1779}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1779","reported_seq":1828,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.170872+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:53.170872+0000","last_peered":"2024-04-26T02:24:53.170872+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:53.170872+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537228,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1861","reported_seq":1899,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.471068+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:54.471068+0000","last_peered":"2024-04-26T02:24:54.471068+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:54.471068+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.832198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15217164,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":14869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1915","reported_seq":1942,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.432737+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:54.432737+0000","last_peered":"2024-04-26T02:24:54.432737+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:54.432737+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15687680,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":15320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'1872","reported_seq":1921,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.191428+0000","last_change":"2024-04-26T02:24:35.636091+0000","last_active":"2024-04-26T02:24:53.191428+0000","last_peered":"2024-04-26T02:24:53.191428+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:53.191428+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1872,"log_dups_size":0,"ondisk_log_size":1872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:57.093134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15282700,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1870,"num_write_kb":14940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1866}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1915","reported_seq":1963,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.179554+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:53.179554+0000","last_peered":"2024-04-26T02:24:53.179554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:53.179554+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15643148,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1883","reported_seq":1922,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.507421+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:54.507421+0000","last_peered":"2024-04-26T02:24:54.507421+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:54.507421+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15352856,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":15017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1795","reported_seq":1855,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:52.532874+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:52.532874+0000","last_peered":"2024-04-26T02:24:52.532874+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:52.532874+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:54.030788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14684684,"num_objects":1793,"num_object_clones":0,"num_object_copies":7172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1793,"num_write_kb":14342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1885","reported_seq":1925,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.514967+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:54.514967+0000","last_peered":"2024-04-26T02:24:54.514967+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:54.514967+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.909483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1861","reported_seq":1884,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.503233+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:54.503233+0000","last_peered":"2024-04-26T02:24:54.503233+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:54.503233+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.913038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15235334,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1860,"num_write_kb":14879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1860","reported_seq":1919,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:52.481491+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:52.481491+0000","last_peered":"2024-04-26T02:24:52.481491+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:52.481491+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.028451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15227142,"num_objects":1859,"num_object_clones":0,"num_object_copies":7436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":14871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1906","reported_seq":1943,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.483983+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:54.483983+0000","last_peered":"2024-04-26T02:24:54.483983+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:54.483983+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1906,"log_dups_size":0,"ondisk_log_size":1906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.875304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15577612,"num_objects":1902,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":15228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1808","reported_seq":1871,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:52.521022+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:52.521022+0000","last_peered":"2024-04-26T02:24:52.521022+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:52.521022+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1808,"log_dups_size":0,"ondisk_log_size":1808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:57.035047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14782988,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1806,"num_write_kb":14445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1805}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1860","reported_seq":1908,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.203579+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:53.203579+0000","last_peered":"2024-04-26T02:24:53.203579+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:53.203579+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15192588,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":14852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242613416,"num_objects":29622,"num_object_clones":0,"num_object_copies":118488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29649,"num_write_kb":237074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362246144,"data_stored":362246144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29676,"ondisk_log_size":29676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":219224,"kb_used_data":119504,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94152616,"statfs":{"total":96636764160,"available":96412278784,"internally_reserved":0,"allocated":122372096,"data_stored":122213558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":102100688},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":224972,"kb_used_data":113284,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94146868,"statfs":{"total":96636764160,"available":96406392832,"internally_reserved":0,"allocated":116002816,"data_stored":115844278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":278528,"data_stored":163207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229092,"kb_used_data":116956,"kb_used_omap":4,"kb_used_meta":112123,"kb_avail":94142748,"statfs":{"total":96636764160,"available":96402173952,"internally_reserved":0,"allocated":119762944,"data_stored":119604406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":114814671},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119996416,"data_stored":119996416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117624832,"data_stored":117624832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124510208,"data_stored":124510208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:57.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:58.277 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:58.291 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-04-26T02:24:58.221649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":715144,"kb_used_data":380768,"kb_used_omap":14,"kb_used_meta":334321,"kb_avail":376772216,"statfs":{"total":386547056640,"available":385814749184,"internally_reserved":0,"allocated":389906432,"data_stored":389302174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":342345267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,10,3,11,29,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42229760,"num_objects":5155,"num_object_clones":0,"num_object_copies":20620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":41240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.789988"},"pg_stats":[{"pgid":"1.f","version":"28'1848","reported_seq":1875,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.484534+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:54.484534+0000","last_peered":"2024-04-26T02:24:54.484534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:54.484534+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1848,"log_dups_size":0,"ondisk_log_size":1848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1846","reported_seq":1895,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.163940+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:53.163940+0000","last_peered":"2024-04-26T02:24:53.163940+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:53.163940+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086092,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":14748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2029","reported_seq":2089,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.500330+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:57.500330+0000","last_peered":"2024-04-26T02:24:57.500330+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:57.500330+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:00.137719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16595206,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":16214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1779","reported_seq":1828,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.170872+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:53.170872+0000","last_peered":"2024-04-26T02:24:53.170872+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:53.170872+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537228,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1861","reported_seq":1899,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.471068+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:54.471068+0000","last_peered":"2024-04-26T02:24:54.471068+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:54.471068+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.832198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15217164,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":14869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1915","reported_seq":1942,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.432737+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:54.432737+0000","last_peered":"2024-04-26T02:24:54.432737+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:54.432737+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15687680,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":15320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'2090","reported_seq":2142,"reported_epoch":29,"state":"peering","last_fresh":"2024-04-26T02:24:57.532047+0000","last_change":"2024-04-26T02:24:57.545822+0000","last_active":"2024-04-26T02:24:57.545822+0000","last_peered":"2024-04-26T02:24:57.532047+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:57.532047+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17068556,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1915","reported_seq":1963,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.179554+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:53.179554+0000","last_peered":"2024-04-26T02:24:53.179554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:53.179554+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15643148,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1883","reported_seq":1922,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.507421+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:54.507421+0000","last_peered":"2024-04-26T02:24:54.507421+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:54.507421+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15352856,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":15017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'2041","reported_seq":2104,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.494517+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:57.494517+0000","last_peered":"2024-04-26T02:24:57.494517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:57.494517+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2041,"log_dups_size":0,"ondisk_log_size":2041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:59.090611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699916,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1885","reported_seq":1925,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.514967+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:54.514967+0000","last_peered":"2024-04-26T02:24:54.514967+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:54.514967+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.909483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1861","reported_seq":1884,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.503233+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:54.503233+0000","last_peered":"2024-04-26T02:24:54.503233+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:54.503233+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.913038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15235334,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1860,"num_write_kb":14879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2113","reported_seq":2173,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.548132+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:57.548132+0000","last_peered":"2024-04-26T02:24:57.548132+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:57.548132+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:01.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17299718,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1906","reported_seq":1943,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.483983+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:54.483983+0000","last_peered":"2024-04-26T02:24:54.483983+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:54.483983+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1906,"log_dups_size":0,"ondisk_log_size":1906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.875304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15577612,"num_objects":1902,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":15228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2122,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.506544+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:57.506544+0000","last_peered":"2024-04-26T02:24:57.506544+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:57.506544+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:02.121410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16822796,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1860","reported_seq":1908,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.203579+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:53.203579+0000","last_peered":"2024-04-26T02:24:53.203579+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:53.203579+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15192588,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":14852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392642560,"data_stored":392642560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":219224,"kb_used_data":119504,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94152616,"statfs":{"total":96636764160,"available":96412278784,"internally_reserved":0,"allocated":122372096,"data_stored":122213558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":102100688},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233244,"kb_used_data":129876,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94138596,"statfs":{"total":96636764160,"available":96397922304,"internally_reserved":0,"allocated":132993024,"data_stored":132830114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,2,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":278528,"data_stored":163207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234940,"kb_used_data":131116,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94136900,"statfs":{"total":96636764160,"available":96396185600,"internally_reserved":0,"allocated":134262784,"data_stored":134095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106294991},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,12,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124510208,"data_stored":124510208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:58.292 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:58.556 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:58.572 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-26T02:24:58.433449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":715144,"kb_used_data":380768,"kb_used_omap":14,"kb_used_meta":334321,"kb_avail":376772216,"statfs":{"total":386547056640,"available":385814749184,"internally_reserved":0,"allocated":389906432,"data_stored":389302174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":342345267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,10,3,11,29,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42229760,"num_objects":5155,"num_object_clones":0,"num_object_copies":20620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":41240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001337"},"pg_stats":[{"pgid":"1.f","version":"28'1848","reported_seq":1875,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.484534+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:54.484534+0000","last_peered":"2024-04-26T02:24:54.484534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:54.484534+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1848,"log_dups_size":0,"ondisk_log_size":1848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1846","reported_seq":1895,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.163940+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:53.163940+0000","last_peered":"2024-04-26T02:24:53.163940+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:53.163940+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086092,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":14748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2029","reported_seq":2089,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.500330+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:57.500330+0000","last_peered":"2024-04-26T02:24:57.500330+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:57.500330+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:00.137719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16595206,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":16214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1779","reported_seq":1828,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.170872+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:53.170872+0000","last_peered":"2024-04-26T02:24:53.170872+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:53.170872+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537228,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1861","reported_seq":1899,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.471068+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:54.471068+0000","last_peered":"2024-04-26T02:24:54.471068+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:54.471068+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.832198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15217164,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":14869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1915","reported_seq":1942,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.432737+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:54.432737+0000","last_peered":"2024-04-26T02:24:54.432737+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:54.432737+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15687680,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":15320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'2090","reported_seq":2142,"reported_epoch":29,"state":"peering","last_fresh":"2024-04-26T02:24:57.532047+0000","last_change":"2024-04-26T02:24:57.545822+0000","last_active":"2024-04-26T02:24:57.545822+0000","last_peered":"2024-04-26T02:24:57.532047+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:57.532047+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17068556,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1915","reported_seq":1963,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.179554+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:53.179554+0000","last_peered":"2024-04-26T02:24:53.179554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:53.179554+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15643148,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1883","reported_seq":1922,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.507421+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:54.507421+0000","last_peered":"2024-04-26T02:24:54.507421+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:54.507421+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15352856,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":15017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'2041","reported_seq":2104,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.494517+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:57.494517+0000","last_peered":"2024-04-26T02:24:57.494517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:57.494517+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2041,"log_dups_size":0,"ondisk_log_size":2041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:59.090611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699916,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1885","reported_seq":1925,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.514967+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:54.514967+0000","last_peered":"2024-04-26T02:24:54.514967+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:54.514967+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.909483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1861","reported_seq":1884,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.503233+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:54.503233+0000","last_peered":"2024-04-26T02:24:54.503233+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:54.503233+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.913038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15235334,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1860,"num_write_kb":14879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2113","reported_seq":2173,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.548132+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:57.548132+0000","last_peered":"2024-04-26T02:24:57.548132+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:57.548132+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:01.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17299718,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1906","reported_seq":1943,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.483983+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:54.483983+0000","last_peered":"2024-04-26T02:24:54.483983+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:54.483983+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1906,"log_dups_size":0,"ondisk_log_size":1906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.875304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15577612,"num_objects":1902,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":15228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2122,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.506544+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:57.506544+0000","last_peered":"2024-04-26T02:24:57.506544+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:57.506544+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:02.121410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16822796,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1860","reported_seq":1908,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.203579+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:53.203579+0000","last_peered":"2024-04-26T02:24:53.203579+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:53.203579+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15192588,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":14852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392642560,"data_stored":392642560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":219224,"kb_used_data":119504,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94152616,"statfs":{"total":96636764160,"available":96412278784,"internally_reserved":0,"allocated":122372096,"data_stored":122213558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":102100688},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233244,"kb_used_data":129876,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94138596,"statfs":{"total":96636764160,"available":96397922304,"internally_reserved":0,"allocated":132993024,"data_stored":132830114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,2,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":278528,"data_stored":163207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234940,"kb_used_data":131116,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94136900,"statfs":{"total":96636764160,"available":96396185600,"internally_reserved":0,"allocated":134262784,"data_stored":134095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106294991},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,12,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124510208,"data_stored":124510208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "28'1848", 'reported_seq': 1875, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.484534+0000', 'last_change': '2024-04-26T02:24:21.690846+0000', 'last_active': '2024-04-26T02:24:54.484534+0000', 'last_peered': '2024-04-26T02:24:54.484534+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:16.519505+0000', 'last_became_peered': '2024-04-26T02:24:16.519505+0000', 'last_unstale': '2024-04-26T02:24:54.484534+0000', 'last_undegraded': '2024-04-26T02:24:21.689455+0000', 'last_fullsized': '2024-04-26T02:24:16.503156+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1848, 'log_dups_size': 0, 'ondisk_log_size': 1848, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15138816, 'num_objects': 1848, 'num_object_clones': 0, 'num_object_copies': 7392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1848, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1848, 'num_write_kb': 14784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "28'1846", 'reported_seq': 1895, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:53.163940+0000', 'last_change': '2024-04-26T02:24:33.022928+0000', 'last_active': '2024-04-26T02:24:53.163940+0000', 'last_peered': '2024-04-26T02:24:53.163940+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:16.521903+0000', 'last_became_peered': '2024-04-26T02:24:16.521903+0000', 'last_unstale': '2024-04-26T02:24:53.163940+0000', 'last_undegraded': '2024-04-26T02:24:16.507064+0000', 'last_fullsized': '2024-04-26T02:24:16.506356+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1846, 'log_dups_size': 0, 'ondisk_log_size': 1846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15086092, 'num_objects': 1842, 'num_object_clones': 0, 'num_object_copies': 7368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1842, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1844, 'num_write_kb': 14748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1842}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "29'2029", 'reported_seq': 2089, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:57.500330+0000', 'last_change': '2024-04-26T02:24:33.606883+0000', 'last_active': '2024-04-26T02:24:57.500330+0000', 'last_peered': '2024-04-26T02:24:57.500330+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:16.516886+0000', 'last_became_peered': '2024-04-26T02:24:16.516886+0000', 'last_unstale': '2024-04-26T02:24:57.500330+0000', 'last_undegraded': '2024-04-26T02:24:16.509359+0000', 'last_fullsized': '2024-04-26T02:24:16.505495+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2029, 'log_dups_size': 0, 'ondisk_log_size': 2029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:25:00.137719+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16595206, 'num_objects': 2026, 'num_object_clones': 0, 'num_object_copies': 8104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2028, 'num_write_kb': 16214, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 2147483647, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2026}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "28'1779", 'reported_seq': 1828, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:53.170872+0000', 'last_change': '2024-04-26T02:24:34.405530+0000', 'last_active': '2024-04-26T02:24:53.170872+0000', 'last_peered': '2024-04-26T02:24:53.170872+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:16.527666+0000', 'last_became_peered': '2024-04-26T02:24:16.527666+0000', 'last_unstale': '2024-04-26T02:24:53.170872+0000', 'last_undegraded': '2024-04-26T02:24:16.509306+0000', 'last_fullsized': '2024-04-26T02:24:16.508519+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1779, 'log_dups_size': 0, 'ondisk_log_size': 1779, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14537228, 'num_objects': 1775, 'num_object_clones': 0, 'num_object_copies': 7100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1775, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1777, 'num_write_kb': 14205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1775}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:24:58.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "28'1861", 'reported_seq': 1899, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.471068+0000', 'last_change': '2024-04-26T02:24:35.212757+0000', 'last_active': '2024-04-26T02:24:54.471068+0000', 'last_peered': '2024-04-26T02:24:54.471068+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:16.526006+0000', 'last_became_peered': '2024-04-26T02:24:16.526006+0000', 'last_unstale': '2024-04-26T02:24:54.471068+0000', 'last_undegraded': '2024-04-26T02:24:16.502444+0000', 'last_fullsized': '2024-04-26T02:24:16.501573+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1861, 'log_dups_size': 0, 'ondisk_log_size': 1861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:58.832198+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15217164, 'num_objects': 1858, 'num_object_clones': 0, 'num_object_copies': 7432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1858, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1859, 'num_write_kb': 14869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2147483647, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1858}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "28'1915", 'reported_seq': 1942, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.432737+0000', 'last_change': '2024-04-26T02:24:21.717835+0000', 'last_active': '2024-04-26T02:24:54.432737+0000', 'last_peered': '2024-04-26T02:24:54.432737+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:16.517484+0000', 'last_became_peered': '2024-04-26T02:24:16.517484+0000', 'last_unstale': '2024-04-26T02:24:54.432737+0000', 'last_undegraded': '2024-04-26T02:24:21.703261+0000', 'last_fullsized': '2024-04-26T02:24:16.502080+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1915, 'log_dups_size': 0, 'ondisk_log_size': 1915, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15687680, 'num_objects': 1915, 'num_object_clones': 0, 'num_object_copies': 7660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1915, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1915, 'num_write_kb': 15320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1915}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "29'2090", 'reported_seq': 2142, 'reported_epoch': 29, 'state': 'peering', 'last_fresh': '2024-04-26T02:24:57.532047+0000', 'last_change': '2024-04-26T02:24:57.545822+0000', 'last_active': '2024-04-26T02:24:57.545822+0000', 'last_peered': '2024-04-26T02:24:57.532047+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:16.523337+0000', 'last_became_peered': '2024-04-26T02:24:16.523337+0000', 'last_unstale': '2024-04-26T02:24:57.532047+0000', 'last_undegraded': '2024-04-26T02:24:16.506528+0000', 'last_fullsized': '2024-04-26T02:24:16.505849+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2090, 'log_dups_size': 0, 'ondisk_log_size': 2090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17068556, 'num_objects': 2084, 'num_object_clones': 0, 'num_object_copies': 8336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2088, 'num_write_kb': 16684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "28'1915", 'reported_seq': 1963, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:53.179554+0000', 'last_change': '2024-04-26T02:24:36.207096+0000', 'last_active': '2024-04-26T02:24:53.179554+0000', 'last_peered': '2024-04-26T02:24:53.179554+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:16.524084+0000', 'last_became_peered': '2024-04-26T02:24:16.524084+0000', 'last_unstale': '2024-04-26T02:24:53.179554+0000', 'last_undegraded': '2024-04-26T02:24:16.509052+0000', 'last_fullsized': '2024-04-26T02:24:16.508425+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1915, 'log_dups_size': 0, 'ondisk_log_size': 1915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15643148, 'num_objects': 1910, 'num_object_clones': 0, 'num_object_copies': 7640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1913, 'num_write_kb': 15292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1910}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "28'1883", 'reported_seq': 1922, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.507421+0000', 'last_change': '2024-04-26T02:24:37.012043+0000', 'last_active': '2024-04-26T02:24:54.507421+0000', 'last_peered': '2024-04-26T02:24:54.507421+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:16.522093+0000', 'last_became_peered': '2024-04-26T02:24:16.522093+0000', 'last_unstale': '2024-04-26T02:24:54.507421+0000', 'last_undegraded': '2024-04-26T02:24:16.502563+0000', 'last_fullsized': '2024-04-26T02:24:16.501655+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1883, 'log_dups_size': 0, 'ondisk_log_size': 1883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15352856, 'num_objects': 1875, 'num_object_clones': 0, 'num_object_copies': 7500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1875, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1880, 'num_write_kb': 15017, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1875}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "29'2041", 'reported_seq': 2104, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:57.494517+0000', 'last_change': '2024-04-26T02:24:37.788140+0000', 'last_active': '2024-04-26T02:24:57.494517+0000', 'last_peered': '2024-04-26T02:24:57.494517+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:16.518378+0000', 'last_became_peered': '2024-04-26T02:24:16.518378+0000', 'last_unstale': '2024-04-26T02:24:57.494517+0000', 'last_undegraded': '2024-04-26T02:24:16.507309+0000', 'last_fullsized': '2024-04-26T02:24:16.504880+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2041, 'log_dups_size': 0, 'ondisk_log_size': 2041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:59.090611+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16699916, 'num_objects': 2039, 'num_object_clones': 0, 'num_object_copies': 8156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2039, 'num_write_kb': 16310, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 2147483647], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2039}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "28'1885", 'reported_seq': 1925, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.514967+0000', 'last_change': '2024-04-26T02:24:39.007164+0000', 'last_active': '2024-04-26T02:24:54.514967+0000', 'last_peered': '2024-04-26T02:24:54.514967+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:16.519989+0000', 'last_became_peered': '2024-04-26T02:24:16.519989+0000', 'last_unstale': '2024-04-26T02:24:54.514967+0000', 'last_undegraded': '2024-04-26T02:24:16.502632+0000', 'last_fullsized': '2024-04-26T02:24:16.501671+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1885, 'log_dups_size': 0, 'ondisk_log_size': 1885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:55.909483+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15395602, 'num_objects': 1880, 'num_object_clones': 0, 'num_object_copies': 7520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1880, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1882, 'num_write_kb': 15044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1880}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "28'1861", 'reported_seq': 1884, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.503233+0000', 'last_change': '2024-04-26T02:24:39.144556+0000', 'last_active': '2024-04-26T02:24:54.503233+0000', 'last_peered': '2024-04-26T02:24:54.503233+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:16.525614+0000', 'last_became_peered': '2024-04-26T02:24:16.525614+0000', 'last_unstale': '2024-04-26T02:24:54.503233+0000', 'last_undegraded': '2024-04-26T02:24:16.504535+0000', 'last_fullsized': '2024-04-26T02:24:16.503837+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1861, 'log_dups_size': 0, 'ondisk_log_size': 1861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:55.913038+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15235334, 'num_objects': 1860, 'num_object_clones': 0, 'num_object_copies': 7440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1860, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1860, 'num_write_kb': 14879, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1860}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "29'2113", 'reported_seq': 2173, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:57.548132+0000', 'last_change': '2024-04-26T02:24:39.939994+0000', 'last_active': '2024-04-26T02:24:57.548132+0000', 'last_peered': '2024-04-26T02:24:57.548132+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:16.528447+0000', 'last_became_peered': '2024-04-26T02:24:16.528447+0000', 'last_unstale': '2024-04-26T02:24:57.548132+0000', 'last_undegraded': '2024-04-26T02:24:16.507269+0000', 'last_fullsized': '2024-04-26T02:24:16.504960+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2113, 'log_dups_size': 0, 'ondisk_log_size': 2113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:25:01.129269+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17299718, 'num_objects': 2112, 'num_object_clones': 0, 'num_object_copies': 8448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2112, 'num_write_kb': 16895, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2112}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "28'1906", 'reported_seq': 1943, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:54.483983+0000', 'last_change': '2024-04-26T02:24:40.773403+0000', 'last_active': '2024-04-26T02:24:54.483983+0000', 'last_peered': '2024-04-26T02:24:54.483983+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:16.519083+0000', 'last_became_peered': '2024-04-26T02:24:16.519083+0000', 'last_unstale': '2024-04-26T02:24:54.483983+0000', 'last_undegraded': '2024-04-26T02:24:16.502632+0000', 'last_fullsized': '2024-04-26T02:24:16.502084+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1906, 'log_dups_size': 0, 'ondisk_log_size': 1906, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:56.875304+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15577612, 'num_objects': 1902, 'num_object_clones': 0, 'num_object_copies': 7608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1904, 'num_write_kb': 15228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1902}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:24:58.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "29'2057", 'reported_seq': 2122, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:57.506544+0000', 'last_change': '2024-04-26T02:24:41.296968+0000', 'last_active': '2024-04-26T02:24:57.506544+0000', 'last_peered': '2024-04-26T02:24:57.506544+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:16.523812+0000', 'last_became_peered': '2024-04-26T02:24:16.523812+0000', 'last_unstale': '2024-04-26T02:24:57.506544+0000', 'last_undegraded': '2024-04-26T02:24:16.508758+0000', 'last_fullsized': '2024-04-26T02:24:16.505050+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2057, 'log_dups_size': 0, 'ondisk_log_size': 2057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:25:02.121410+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16822796, 'num_objects': 2054, 'num_object_clones': 0, 'num_object_copies': 8216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2055, 'num_write_kb': 16437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2054}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:24:58.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:24:58.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "28'1860", 'reported_seq': 1908, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-26T02:24:53.203579+0000', 'last_change': '2024-04-26T02:24:42.084358+0000', 'last_active': '2024-04-26T02:24:53.203579+0000', 'last_peered': '2024-04-26T02:24:53.203579+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:16.523238+0000', 'last_became_peered': '2024-04-26T02:24:16.523238+0000', 'last_unstale': '2024-04-26T02:24:53.203579+0000', 'last_undegraded': '2024-04-26T02:24:16.507802+0000', 'last_fullsized': '2024-04-26T02:24:16.506713+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1860, 'log_dups_size': 0, 'ondisk_log_size': 1860, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15192588, 'num_objects': 1855, 'num_object_clones': 0, 'num_object_copies': 7420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1855, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1858, 'num_write_kb': 14852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1855}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:24:58.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:24:58.689 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 37 20 32720 32700 6.902 0 - 0.0219152 2024-04-26T02:24:58.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:58.896 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29622/118488 objects degraded (25.000%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":95,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":16}],"num_pgs":16,"num_pools":1,"num_objects":29622,"data_bytes":242613416,"bytes_used":717291520,"bytes_avail":385829765120,"bytes_total":386547056640,"degraded_objects":29622,"degraded_total":118488,"degraded_ratio":0.25,"write_bytes_sec":5647843,"read_op_per_sec":0,"write_op_per_sec":689},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-04-26T02:24:34.427098+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:24:58.912 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:24:58.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:24:59.201 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:24:59.201 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:24:59.218 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-26T02:24:58.433449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":715144,"kb_used_data":380768,"kb_used_omap":14,"kb_used_meta":334321,"kb_avail":376772216,"statfs":{"total":386547056640,"available":385814749184,"internally_reserved":0,"allocated":389906432,"data_stored":389302174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":342345267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,10,3,11,29,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42229760,"num_objects":5155,"num_object_clones":0,"num_object_copies":20620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":41240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001337"},"pg_stats":[{"pgid":"1.f","version":"28'1848","reported_seq":1875,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.484534+0000","last_change":"2024-04-26T02:24:21.690846+0000","last_active":"2024-04-26T02:24:54.484534+0000","last_peered":"2024-04-26T02:24:54.484534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:54.484534+0000","last_undegraded":"2024-04-26T02:24:21.689455+0000","last_fullsized":"2024-04-26T02:24:16.503156+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1848,"log_dups_size":0,"ondisk_log_size":1848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:59.522398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1846","reported_seq":1895,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.163940+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:53.163940+0000","last_peered":"2024-04-26T02:24:53.163940+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:53.163940+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086092,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":14748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2029","reported_seq":2089,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.500330+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:57.500330+0000","last_peered":"2024-04-26T02:24:57.500330+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:57.500330+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:00.137719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16595206,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":16214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1779","reported_seq":1828,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.170872+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:53.170872+0000","last_peered":"2024-04-26T02:24:53.170872+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:53.170872+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537228,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'1861","reported_seq":1899,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.471068+0000","last_change":"2024-04-26T02:24:35.212757+0000","last_active":"2024-04-26T02:24:54.471068+0000","last_peered":"2024-04-26T02:24:54.471068+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:54.471068+0000","last_undegraded":"2024-04-26T02:24:16.502444+0000","last_fullsized":"2024-04-26T02:24:16.501573+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.832198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15217164,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":14869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'1915","reported_seq":1942,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.432737+0000","last_change":"2024-04-26T02:24:21.717835+0000","last_active":"2024-04-26T02:24:54.432737+0000","last_peered":"2024-04-26T02:24:54.432737+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:54.432737+0000","last_undegraded":"2024-04-26T02:24:21.703261+0000","last_fullsized":"2024-04-26T02:24:16.502080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:53:30.621647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15687680,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":15320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'2090","reported_seq":2142,"reported_epoch":29,"state":"peering","last_fresh":"2024-04-26T02:24:57.532047+0000","last_change":"2024-04-26T02:24:57.545822+0000","last_active":"2024-04-26T02:24:57.545822+0000","last_peered":"2024-04-26T02:24:57.532047+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:57.532047+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17068556,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1915","reported_seq":1963,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.179554+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:53.179554+0000","last_peered":"2024-04-26T02:24:53.179554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:53.179554+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15643148,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1883","reported_seq":1922,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.507421+0000","last_change":"2024-04-26T02:24:37.012043+0000","last_active":"2024-04-26T02:24:54.507421+0000","last_peered":"2024-04-26T02:24:54.507421+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:16.522093+0000","last_became_peered":"2024-04-26T02:24:16.522093+0000","last_unstale":"2024-04-26T02:24:54.507421+0000","last_undegraded":"2024-04-26T02:24:16.502563+0000","last_fullsized":"2024-04-26T02:24:16.501655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15352856,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":15017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'2041","reported_seq":2104,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.494517+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:57.494517+0000","last_peered":"2024-04-26T02:24:57.494517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:57.494517+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2041,"log_dups_size":0,"ondisk_log_size":2041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:59.090611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699916,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'1885","reported_seq":1925,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.514967+0000","last_change":"2024-04-26T02:24:39.007164+0000","last_active":"2024-04-26T02:24:54.514967+0000","last_peered":"2024-04-26T02:24:54.514967+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:54.514967+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.501671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.909483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"28'1861","reported_seq":1884,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.503233+0000","last_change":"2024-04-26T02:24:39.144556+0000","last_active":"2024-04-26T02:24:54.503233+0000","last_peered":"2024-04-26T02:24:54.503233+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:54.503233+0000","last_undegraded":"2024-04-26T02:24:16.504535+0000","last_fullsized":"2024-04-26T02:24:16.503837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.913038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15235334,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1860,"num_write_kb":14879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2113","reported_seq":2173,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.548132+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:57.548132+0000","last_peered":"2024-04-26T02:24:57.548132+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:57.548132+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:01.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17299718,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1906","reported_seq":1943,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:54.483983+0000","last_change":"2024-04-26T02:24:40.773403+0000","last_active":"2024-04-26T02:24:54.483983+0000","last_peered":"2024-04-26T02:24:54.483983+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:54.483983+0000","last_undegraded":"2024-04-26T02:24:16.502632+0000","last_fullsized":"2024-04-26T02:24:16.502084+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1906,"log_dups_size":0,"ondisk_log_size":1906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.875304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15577612,"num_objects":1902,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":15228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2122,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.506544+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:57.506544+0000","last_peered":"2024-04-26T02:24:57.506544+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:57.506544+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:02.121410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16822796,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1860","reported_seq":1908,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.203579+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:53.203579+0000","last_peered":"2024-04-26T02:24:53.203579+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:53.203579+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15192588,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":14852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":252550312,"num_objects":30835,"num_object_clones":0,"num_object_copies":123340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30862,"num_write_kb":246778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392642560,"data_stored":392642560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30889,"ondisk_log_size":30889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":219224,"kb_used_data":119504,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94152616,"statfs":{"total":96636764160,"available":96412278784,"internally_reserved":0,"allocated":122372096,"data_stored":122213558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":102100688},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233244,"kb_used_data":129876,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94138596,"statfs":{"total":96636764160,"available":96397922304,"internally_reserved":0,"allocated":132993024,"data_stored":132830114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,2,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":278528,"data_stored":163207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234940,"kb_used_data":131116,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94136900,"statfs":{"total":96636764160,"available":96396185600,"internally_reserved":0,"allocated":134262784,"data_stored":134095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106294991},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,12,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124510208,"data_stored":124510208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:24:59.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 38 20 32761 32741 6.72885 0.160156 0.0434566 0.023129 2024-04-26T02:25:00.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 39 20 33174 33154 6.63906 3.22656 0.0478744 0.0235166 2024-04-26T02:25:01.690 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:25:01.692987+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0238124 2024-04-26T02:25:01.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:25:01.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 40 20 33598 33578 6.55581 3.3125 0.0148474 0.0238124 2024-04-26T02:25:02.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:02.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 41 20 33999 33979 6.47221 3.13281 0.120094 0.0241163 2024-04-26T02:25:03.376 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:03.377 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:03.491 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-04-26T02:25:02.434322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":261012648,"num_objects":31868,"num_object_clones":0,"num_object_copies":127472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31895,"num_write_kb":255042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31922,"ondisk_log_size":31922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":755952,"kb_used_data":395192,"kb_used_omap":14,"kb_used_meta":360689,"kb_avail":376731408,"statfs":{"total":386547056640,"available":385772961792,"internally_reserved":0,"allocated":404676608,"data_stored":403998820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":369346099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,3,13,22,4,0,0,0,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18399232,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":17968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.913908"},"pg_stats":[{"pgid":"1.f","version":"29'2017","reported_seq":2050,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320737+0000","last_change":"2024-04-26T02:24:59.320737+0000","last_active":"2024-04-26T02:24:57.543253+0000","last_peered":"2024-04-26T02:24:57.519322+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:59.320737+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16523264,"num_objects":2017,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2017,"num_write_kb":16136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2017}],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1846","reported_seq":1895,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.163940+0000","last_change":"2024-04-26T02:24:33.022928+0000","last_active":"2024-04-26T02:24:53.163940+0000","last_peered":"2024-04-26T02:24:53.163940+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:16.521903+0000","last_became_peered":"2024-04-26T02:24:16.521903+0000","last_unstale":"2024-04-26T02:24:53.163940+0000","last_undegraded":"2024-04-26T02:24:16.507064+0000","last_fullsized":"2024-04-26T02:24:16.506356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:55.059374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15086092,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":14748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2029","reported_seq":2089,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.500330+0000","last_change":"2024-04-26T02:24:33.606883+0000","last_active":"2024-04-26T02:24:57.500330+0000","last_peered":"2024-04-26T02:24:57.500330+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:16.516886+0000","last_became_peered":"2024-04-26T02:24:16.516886+0000","last_unstale":"2024-04-26T02:24:57.500330+0000","last_undegraded":"2024-04-26T02:24:16.509359+0000","last_fullsized":"2024-04-26T02:24:16.505495+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:00.137719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16595206,"num_objects":2026,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2028,"num_write_kb":16214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1779","reported_seq":1828,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.170872+0000","last_change":"2024-04-26T02:24:34.405530+0000","last_active":"2024-04-26T02:24:53.170872+0000","last_peered":"2024-04-26T02:24:53.170872+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:16.527666+0000","last_became_peered":"2024-04-26T02:24:16.527666+0000","last_unstale":"2024-04-26T02:24:53.170872+0000","last_undegraded":"2024-04-26T02:24:16.509306+0000","last_fullsized":"2024-04-26T02:24:16.508519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:56.079940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537228,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":14205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'2016","reported_seq":2063,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.322153+0000","last_change":"2024-04-26T02:24:59.322153+0000","last_active":"2024-04-26T02:24:57.538954+0000","last_peered":"2024-04-26T02:24:57.529629+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:59.322153+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2016,"log_dups_size":0,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16486924,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":16109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'2051","reported_seq":2086,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.321894+0000","last_change":"2024-04-26T02:24:59.321894+0000","last_active":"2024-04-26T02:24:57.541874+0000","last_peered":"2024-04-26T02:24:57.508700+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:59.321894+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16801792,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":16408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2051}],"blocked_by":[3],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"29'2090","reported_seq":2142,"reported_epoch":29,"state":"peering","last_fresh":"2024-04-26T02:24:57.532047+0000","last_change":"2024-04-26T02:24:57.545822+0000","last_active":"2024-04-26T02:24:57.545822+0000","last_peered":"2024-04-26T02:24:57.532047+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:16.523337+0000","last_became_peered":"2024-04-26T02:24:16.523337+0000","last_unstale":"2024-04-26T02:24:57.532047+0000","last_undegraded":"2024-04-26T02:24:16.506528+0000","last_fullsized":"2024-04-26T02:24:16.505849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17068556,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":16684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1915","reported_seq":1963,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.179554+0000","last_change":"2024-04-26T02:24:36.207096+0000","last_active":"2024-04-26T02:24:53.179554+0000","last_peered":"2024-04-26T02:24:53.179554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:16.524084+0000","last_became_peered":"2024-04-26T02:24:16.524084+0000","last_unstale":"2024-04-26T02:24:53.179554+0000","last_undegraded":"2024-04-26T02:24:16.509052+0000","last_fullsized":"2024-04-26T02:24:16.508425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15643148,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2016","reported_seq":2068,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.564623+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:24:59.564623+0000","last_peered":"2024-04-26T02:24:59.564623+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:24:59.564623+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2016,"log_dups_size":0,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16442392,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":16081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"29'2041","reported_seq":2104,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.494517+0000","last_change":"2024-04-26T02:24:37.788140+0000","last_active":"2024-04-26T02:24:57.494517+0000","last_peered":"2024-04-26T02:24:57.494517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:16.518378+0000","last_became_peered":"2024-04-26T02:24:16.518378+0000","last_unstale":"2024-04-26T02:24:57.494517+0000","last_undegraded":"2024-04-26T02:24:16.507309+0000","last_fullsized":"2024-04-26T02:24:16.504880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2041,"log_dups_size":0,"ondisk_log_size":2041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:59.090611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699916,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":16310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2039}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'2025","reported_seq":2074,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320888+0000","last_change":"2024-04-26T02:24:59.320888+0000","last_active":"2024-04-26T02:24:57.542843+0000","last_peered":"2024-04-26T02:24:57.490760+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:59.320888+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16542482,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"29'1990","reported_seq":2020,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320905+0000","last_change":"2024-04-26T02:24:59.320905+0000","last_active":"2024-04-26T02:24:58.243684+0000","last_peered":"2024-04-26T02:24:57.520704+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:59.320905+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1990,"log_dups_size":0,"ondisk_log_size":1990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16292102,"num_objects":1989,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":15911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1}],"blocked_by":[0],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2113","reported_seq":2173,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.548132+0000","last_change":"2024-04-26T02:24:39.939994+0000","last_active":"2024-04-26T02:24:57.548132+0000","last_peered":"2024-04-26T02:24:57.548132+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:16.528447+0000","last_became_peered":"2024-04-26T02:24:16.528447+0000","last_unstale":"2024-04-26T02:24:57.548132+0000","last_undegraded":"2024-04-26T02:24:16.507269+0000","last_fullsized":"2024-04-26T02:24:16.504960+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:01.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17299718,"num_objects":2112,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":16895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"29'2077","reported_seq":2121,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320918+0000","last_change":"2024-04-26T02:24:59.320918+0000","last_active":"2024-04-26T02:24:58.239161+0000","last_peered":"2024-04-26T02:24:57.535535+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:59.320918+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16978444,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2075,"num_write_kb":16596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2}],"blocked_by":[0],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"29'2057","reported_seq":2122,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:57.506544+0000","last_change":"2024-04-26T02:24:41.296968+0000","last_active":"2024-04-26T02:24:57.506544+0000","last_peered":"2024-04-26T02:24:57.506544+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:16.523812+0000","last_became_peered":"2024-04-26T02:24:16.523812+0000","last_unstale":"2024-04-26T02:24:57.506544+0000","last_undegraded":"2024-04-26T02:24:16.508758+0000","last_fullsized":"2024-04-26T02:24:16.505050+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:02.121410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16822796,"num_objects":2054,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":16437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'1860","reported_seq":1908,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-04-26T02:24:53.203579+0000","last_change":"2024-04-26T02:24:42.084358+0000","last_active":"2024-04-26T02:24:53.203579+0000","last_peered":"2024-04-26T02:24:53.203579+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:16.523238+0000","last_became_peered":"2024-04-26T02:24:16.523238+0000","last_unstale":"2024-04-26T02:24:53.203579+0000","last_undegraded":"2024-04-26T02:24:16.507802+0000","last_fullsized":"2024-04-26T02:24:16.506713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:58.094025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15192588,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":14852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":261012648,"num_objects":31868,"num_object_clones":0,"num_object_copies":127472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31895,"num_write_kb":255042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405659648,"data_stored":405659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31922,"ondisk_log_size":31922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235012,"kb_used_data":131120,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94136828,"statfs":{"total":96636764160,"available":96396111872,"internally_reserved":0,"allocated":134266880,"data_stored":134095454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":106360528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233244,"kb_used_data":129876,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94138596,"statfs":{"total":96636764160,"available":96397922304,"internally_reserved":0,"allocated":132993024,"data_stored":132830114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,2,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52756,"kb_used_data":3080,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3153920,"data_stored":2977957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234940,"kb_used_data":131116,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94136900,"statfs":{"total":96636764160,"available":96396185600,"internally_reserved":0,"allocated":134262784,"data_stored":134095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106294991},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,12,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3620864,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134008832,"data_stored":134008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134021120,"data_stored":134021120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:03.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:03.691 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 42 20 34384 34364 6.38975 3.00781 0.0768807 0.0244271 2024-04-26T02:25:04.548 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:04.548 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:04.655 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-04-26T02:25:04.436810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274447528,"num_objects":33508,"num_object_clones":0,"num_object_copies":134032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33535,"num_write_kb":268162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33562,"ondisk_log_size":33562,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":775544,"kb_used_data":406472,"kb_used_omap":14,"kb_used_meta":369009,"kb_avail":376711816,"statfs":{"total":386547056640,"available":385752899584,"internally_reserved":0,"allocated":416227328,"data_stored":415528013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":377865779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,13,3,2,5,0,0,6,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31834112,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":31088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.910755"},"pg_stats":[{"pgid":"1.f","version":"29'2017","reported_seq":2050,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320737+0000","last_change":"2024-04-26T02:24:59.320737+0000","last_active":"2024-04-26T02:24:57.543253+0000","last_peered":"2024-04-26T02:24:57.519322+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.519505+0000","last_became_peered":"2024-04-26T02:24:16.519505+0000","last_unstale":"2024-04-26T02:24:59.320737+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16523264,"num_objects":2017,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2017,"num_write_kb":16136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2017}],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2170","reported_seq":2231,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.085661+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:03.085661+0000","last_peered":"2024-04-26T02:25:03.085661+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:03.085661+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2170,"log_dups_size":0,"ondisk_log_size":2170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17740300,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":90}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2110","reported_seq":2183,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.403954+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:02.403954+0000","last_peered":"2024-04-26T02:25:02.403954+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:02.403954+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2110,"log_dups_size":0,"ondisk_log_size":2110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17258758,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":16862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":83}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2072","reported_seq":2129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.160870+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:03.160870+0000","last_peered":"2024-04-26T02:25:03.160870+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:03.160870+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2072,"log_dups_size":0,"ondisk_log_size":2072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937484,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":16549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"29'2016","reported_seq":2063,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.322153+0000","last_change":"2024-04-26T02:24:59.322153+0000","last_active":"2024-04-26T02:24:57.538954+0000","last_peered":"2024-04-26T02:24:57.529629+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:16.526006+0000","last_became_peered":"2024-04-26T02:24:16.526006+0000","last_unstale":"2024-04-26T02:24:59.322153+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2016,"log_dups_size":0,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16486924,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":16109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2}],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"29'2051","reported_seq":2086,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.321894+0000","last_change":"2024-04-26T02:24:59.321894+0000","last_active":"2024-04-26T02:24:57.541874+0000","last_peered":"2024-04-26T02:24:57.508700+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:16.517484+0000","last_became_peered":"2024-04-26T02:24:16.517484+0000","last_unstale":"2024-04-26T02:24:59.321894+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16801792,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":16408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2051}],"blocked_by":[3],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2164","reported_seq":2349,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.215088+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:03.215088+0000","last_peered":"2024-04-26T02:25:03.215088+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:03.215088+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17674764,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":17276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2251","reported_seq":2312,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.212607+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:03.212607+0000","last_peered":"2024-04-26T02:25:03.212607+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:03.212607+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18395660,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2016","reported_seq":2068,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.564623+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:24:59.564623+0000","last_peered":"2024-04-26T02:24:59.564623+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:24:59.564623+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2016,"log_dups_size":0,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16442392,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":16081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2109","reported_seq":2183,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.511592+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:02.511592+0000","last_peered":"2024-04-26T02:25:02.511592+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:02.511592+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17256972,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2107,"num_write_kb":16854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"29'2025","reported_seq":2074,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320888+0000","last_change":"2024-04-26T02:24:59.320888+0000","last_active":"2024-04-26T02:24:57.542843+0000","last_peered":"2024-04-26T02:24:57.490760+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:16.519989+0000","last_became_peered":"2024-04-26T02:24:16.519989+0000","last_unstale":"2024-04-26T02:24:59.320888+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16542482,"num_objects":2020,"num_object_clones":0,"num_object_copies":8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"29'1990","reported_seq":2020,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320905+0000","last_change":"2024-04-26T02:24:59.320905+0000","last_active":"2024-04-26T02:24:58.243684+0000","last_peered":"2024-04-26T02:24:57.520704+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:16.525614+0000","last_became_peered":"2024-04-26T02:24:16.525614+0000","last_unstale":"2024-04-26T02:24:59.320905+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":1990,"log_dups_size":0,"ondisk_log_size":1990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16292102,"num_objects":1989,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":15911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1}],"blocked_by":[0],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2196","reported_seq":2266,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.537557+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:02.537557+0000","last_peered":"2024-04-26T02:25:02.537557+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:02.537557+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17979654,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"29'2077","reported_seq":2121,"reported_epoch":32,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-26T02:24:59.320918+0000","last_change":"2024-04-26T02:24:59.320918+0000","last_active":"2024-04-26T02:24:58.239161+0000","last_peered":"2024-04-26T02:24:57.535535+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:16.519083+0000","last_became_peered":"2024-04-26T02:24:16.519083+0000","last_unstale":"2024-04-26T02:24:59.320918+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16978444,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2075,"num_write_kb":16596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2}],"blocked_by":[0],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2136","reported_seq":2210,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.535542+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:02.535542+0000","last_peered":"2024-04-26T02:25:02.535542+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:02.535542+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17469964,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":17069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":80}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2162","reported_seq":2221,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.117671+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:03.117671+0000","last_peered":"2024-04-26T02:25:03.117671+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:03.117671+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17666572,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":93}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274447528,"num_objects":33508,"num_object_clones":0,"num_object_copies":134032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33535,"num_write_kb":268162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":416772096,"data_stored":416772096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33562,"ondisk_log_size":33562,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235012,"kb_used_data":131120,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94136828,"statfs":{"total":96636764160,"available":96396111872,"internally_reserved":0,"allocated":134266880,"data_stored":134095454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":106360528},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243000,"kb_used_data":135472,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94128840,"statfs":{"total":96636764160,"available":96387932160,"internally_reserved":0,"allocated":138723328,"data_stored":138547365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52756,"kb_used_data":3080,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3153920,"data_stored":2977957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244776,"kb_used_data":136800,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94127064,"statfs":{"total":96636764160,"available":96386113536,"internally_reserved":0,"allocated":140083200,"data_stored":139907237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,8,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140091392,"data_stored":140091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3620864,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139038720,"data_stored":139038720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134021120,"data_stored":134021120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:04.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "29'2017", 'reported_seq': 2050, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.320737+0000', 'last_change': '2024-04-26T02:24:59.320737+0000', 'last_active': '2024-04-26T02:24:57.543253+0000', 'last_peered': '2024-04-26T02:24:57.519322+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:16.519505+0000', 'last_became_peered': '2024-04-26T02:24:16.519505+0000', 'last_unstale': '2024-04-26T02:24:59.320737+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2017, 'log_dups_size': 0, 'ondisk_log_size': 2017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16523264, 'num_objects': 2017, 'num_object_clones': 0, 'num_object_copies': 8068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2017, 'num_write_kb': 16136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2017}], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2170", 'reported_seq': 2231, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:03.085661+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:03.085661+0000', 'last_peered': '2024-04-26T02:25:03.085661+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:03.085661+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2170, 'log_dups_size': 0, 'ondisk_log_size': 2170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17740300, 'num_objects': 2166, 'num_object_clones': 0, 'num_object_copies': 8664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2168, 'num_write_kb': 17340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 90}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2110", 'reported_seq': 2183, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:02.403954+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:02.403954+0000', 'last_peered': '2024-04-26T02:25:02.403954+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:02.403954+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2110, 'log_dups_size': 0, 'ondisk_log_size': 2110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17258758, 'num_objects': 2107, 'num_object_clones': 0, 'num_object_copies': 8428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2109, 'num_write_kb': 16862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 83}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2072", 'reported_seq': 2129, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:03.160870+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:03.160870+0000', 'last_peered': '2024-04-26T02:25:03.160870+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:03.160870+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2072, 'log_dups_size': 0, 'ondisk_log_size': 2072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16937484, 'num_objects': 2068, 'num_object_clones': 0, 'num_object_copies': 8272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2070, 'num_write_kb': 16549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:04.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "29'2016", 'reported_seq': 2063, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.322153+0000', 'last_change': '2024-04-26T02:24:59.322153+0000', 'last_active': '2024-04-26T02:24:57.538954+0000', 'last_peered': '2024-04-26T02:24:57.529629+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:16.526006+0000', 'last_became_peered': '2024-04-26T02:24:16.526006+0000', 'last_unstale': '2024-04-26T02:24:59.322153+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2016, 'log_dups_size': 0, 'ondisk_log_size': 2016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16486924, 'num_objects': 2013, 'num_object_clones': 0, 'num_object_copies': 8052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2014, 'num_write_kb': 16109, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2}], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "29'2051", 'reported_seq': 2086, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.321894+0000', 'last_change': '2024-04-26T02:24:59.321894+0000', 'last_active': '2024-04-26T02:24:57.541874+0000', 'last_peered': '2024-04-26T02:24:57.508700+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:16.517484+0000', 'last_became_peered': '2024-04-26T02:24:16.517484+0000', 'last_unstale': '2024-04-26T02:24:59.321894+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2051, 'log_dups_size': 0, 'ondisk_log_size': 2051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16801792, 'num_objects': 2051, 'num_object_clones': 0, 'num_object_copies': 8204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2051, 'num_write_kb': 16408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2051}], 'blocked_by': [3], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2164", 'reported_seq': 2349, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:03.215088+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:03.215088+0000', 'last_peered': '2024-04-26T02:25:03.215088+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:03.215088+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2164, 'log_dups_size': 0, 'ondisk_log_size': 2164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17674764, 'num_objects': 2158, 'num_object_clones': 0, 'num_object_copies': 8632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2021, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2162, 'num_write_kb': 17276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 499712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 137}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2251", 'reported_seq': 2312, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:03.212607+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:03.212607+0000', 'last_peered': '2024-04-26T02:25:03.212607+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:03.212607+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2251, 'log_dups_size': 0, 'ondisk_log_size': 2251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18395660, 'num_objects': 2246, 'num_object_clones': 0, 'num_object_copies': 8984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2249, 'num_write_kb': 17980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:04.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "29'2016", 'reported_seq': 2068, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.564623+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:24:59.564623+0000', 'last_peered': '2024-04-26T02:24:59.564623+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:24:59.564623+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2016, 'log_dups_size': 0, 'ondisk_log_size': 2016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16442392, 'num_objects': 2008, 'num_object_clones': 0, 'num_object_copies': 8032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2013, 'num_write_kb': 16081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2109", 'reported_seq': 2183, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:02.511592+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:02.511592+0000', 'last_peered': '2024-04-26T02:25:02.511592+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:02.511592+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2109, 'log_dups_size': 0, 'ondisk_log_size': 2109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17256972, 'num_objects': 2107, 'num_object_clones': 0, 'num_object_copies': 8428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2107, 'num_write_kb': 16854, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 70}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "29'2025", 'reported_seq': 2074, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.320888+0000', 'last_change': '2024-04-26T02:24:59.320888+0000', 'last_active': '2024-04-26T02:24:57.542843+0000', 'last_peered': '2024-04-26T02:24:57.490760+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:16.519989+0000', 'last_became_peered': '2024-04-26T02:24:16.519989+0000', 'last_unstale': '2024-04-26T02:24:59.320888+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2025, 'log_dups_size': 0, 'ondisk_log_size': 2025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16542482, 'num_objects': 2020, 'num_object_clones': 0, 'num_object_copies': 8080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2022, 'num_write_kb': 16164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3}], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "29'1990", 'reported_seq': 2020, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.320905+0000', 'last_change': '2024-04-26T02:24:59.320905+0000', 'last_active': '2024-04-26T02:24:58.243684+0000', 'last_peered': '2024-04-26T02:24:57.520704+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:16.525614+0000', 'last_became_peered': '2024-04-26T02:24:16.525614+0000', 'last_unstale': '2024-04-26T02:24:59.320905+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 1990, 'log_dups_size': 0, 'ondisk_log_size': 1990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16292102, 'num_objects': 1989, 'num_object_clones': 0, 'num_object_copies': 7956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1989, 'num_write_kb': 15911, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1}], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2196", 'reported_seq': 2266, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:02.537557+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:02.537557+0000', 'last_peered': '2024-04-26T02:25:02.537557+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:02.537557+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2196, 'log_dups_size': 0, 'ondisk_log_size': 2196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17979654, 'num_objects': 2195, 'num_object_clones': 0, 'num_object_copies': 8780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2195, 'num_write_kb': 17559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 84}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:04.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "29'2077", 'reported_seq': 2121, 'reported_epoch': 32, 'state': 'activating+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:24:59.320918+0000', 'last_change': '2024-04-26T02:24:59.320918+0000', 'last_active': '2024-04-26T02:24:58.239161+0000', 'last_peered': '2024-04-26T02:24:57.535535+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:16.519083+0000', 'last_became_peered': '2024-04-26T02:24:16.519083+0000', 'last_unstale': '2024-04-26T02:24:59.320918+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2077, 'log_dups_size': 0, 'ondisk_log_size': 2077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16978444, 'num_objects': 2073, 'num_object_clones': 0, 'num_object_copies': 8292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2075, 'num_write_kb': 16596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2}], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:04.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:04.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2136", 'reported_seq': 2210, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:02.535542+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:02.535542+0000', 'last_peered': '2024-04-26T02:25:02.535542+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:02.535542+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2136, 'log_dups_size': 0, 'ondisk_log_size': 2136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17469964, 'num_objects': 2133, 'num_object_clones': 0, 'num_object_copies': 8532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2134, 'num_write_kb': 17069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 80}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:04.660 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:04.660 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2162", 'reported_seq': 2221, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:03.117671+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:03.117671+0000', 'last_peered': '2024-04-26T02:25:03.117671+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:03.117671+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2162, 'log_dups_size': 0, 'ondisk_log_size': 2162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17666572, 'num_objects': 2157, 'num_object_clones': 0, 'num_object_copies': 8628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2160, 'num_write_kb': 17268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 93}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:04.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:04.694 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 43 20 34771 34751 6.31133 3.02344 0.0189809 0.0247304 2024-04-26T02:25:05.693 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 44 20 35149 35129 6.23503 2.95312 0.0592548 0.0250348 2024-04-26T02:25:06.029 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:06.030 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32679/134032 objects degraded (24.381%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"activating+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33508,"data_bytes":274447528,"bytes_used":794157056,"bytes_avail":385752899584,"bytes_total":386547056640,"inactive_pgs_ratio":0.375,"degraded_objects":32679,"degraded_total":134032,"degraded_ratio":0.24381490987226934,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":72309,"recovering_keys_per_sec":0,"num_objects_recovered":61,"num_bytes_recovered":499712,"num_keys_recovered":0,"write_bytes_sec":4606458,"read_op_per_sec":0,"write_op_per_sec":562},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-04-26T02:24:34.427098+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:06.122 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:06.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:06.693 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 45 20 35521 35501 6.16107 2.90625 0.0260022 0.0253379 2024-04-26T02:25:07.564 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:07.565 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:07.619 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-26T02:25:06.437298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":281672872,"num_objects":34390,"num_object_clones":0,"num_object_copies":137560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34417,"num_write_kb":275218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34444,"ondisk_log_size":34444,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":808716,"kb_used_data":413288,"kb_used_omap":14,"kb_used_meta":395377,"kb_avail":376678644,"statfs":{"total":386547056640,"available":385718931456,"internally_reserved":0,"allocated":423206912,"data_stored":422503060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":404866611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,15,7,4,10,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3555,"num_write_kb":28440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.215648"},"pg_stats":[{"pgid":"1.f","version":"32'2145","reported_seq":2181,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.588044+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:04.588044+0000","last_peered":"2024-04-26T02:25:04.588044+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:04.588044+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17571840,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2170","reported_seq":2231,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.085661+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:03.085661+0000","last_peered":"2024-04-26T02:25:03.085661+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:03.085661+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2170,"log_dups_size":0,"ondisk_log_size":2170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17740300,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":90}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2110","reported_seq":2183,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.403954+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:02.403954+0000","last_peered":"2024-04-26T02:25:02.403954+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:02.403954+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2110,"log_dups_size":0,"ondisk_log_size":2110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17258758,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":16862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":83}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2072","reported_seq":2129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.160870+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:03.160870+0000","last_peered":"2024-04-26T02:25:03.160870+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:03.160870+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2072,"log_dups_size":0,"ondisk_log_size":2072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937484,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":16549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2149","reported_seq":2199,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.574792+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:04.574792+0000","last_peered":"2024-04-26T02:25:04.574792+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:04.574792+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17576460,"num_objects":2146,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":17173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2166","reported_seq":2204,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.606645+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:04.606645+0000","last_peered":"2024-04-26T02:25:04.606645+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:04.606645+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2166,"log_dups_size":0,"ondisk_log_size":2166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17743872,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2166,"num_write_kb":17328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2164","reported_seq":2349,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.215088+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:03.215088+0000","last_peered":"2024-04-26T02:25:03.215088+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:03.215088+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17674764,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":17276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2251","reported_seq":2312,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.212607+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:03.212607+0000","last_peered":"2024-04-26T02:25:03.212607+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:03.212607+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18395660,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2149","reported_seq":2201,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.541013+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:04.541013+0000","last_peered":"2024-04-26T02:25:04.541013+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:04.541013+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17531928,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":17145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":137}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2109","reported_seq":2183,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.511592+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:02.511592+0000","last_peered":"2024-04-26T02:25:02.511592+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:02.511592+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17256972,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2107,"num_write_kb":16854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2136","reported_seq":2189,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.452330+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:04.452330+0000","last_peered":"2024-04-26T02:25:04.452330+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:04.452330+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17451794,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2133,"num_write_kb":17052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2112","reported_seq":2146,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.499459+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:04.499459+0000","last_peered":"2024-04-26T02:25:04.499459+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:04.499459+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17291526,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":16887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2196","reported_seq":2266,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.537557+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:02.537557+0000","last_peered":"2024-04-26T02:25:02.537557+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:02.537557+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17979654,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2217","reported_seq":2264,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:04.578714+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:04.578714+0000","last_peered":"2024-04-26T02:25:04.578714+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:04.578714+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2217,"log_dups_size":0,"ondisk_log_size":2217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18125324,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2215,"num_write_kb":17716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2136","reported_seq":2210,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:02.535542+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:02.535542+0000","last_peered":"2024-04-26T02:25:02.535542+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:02.535542+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17469964,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":17069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":80}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2162","reported_seq":2221,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:03.117671+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:03.117671+0000","last_peered":"2024-04-26T02:25:03.117671+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:03.117671+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17666572,"num_objects":2157,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":93}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":281672872,"num_objects":34390,"num_object_clones":0,"num_object_copies":137560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34417,"num_write_kb":275218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":499712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425070592,"data_stored":425070592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34444,"ondisk_log_size":34444,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":268184,"kb_used_data":137936,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94103656,"statfs":{"total":96636764160,"available":96362143744,"internally_reserved":0,"allocated":141246464,"data_stored":141070501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":133361360},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243000,"kb_used_data":135472,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94128840,"statfs":{"total":96636764160,"available":96387932160,"internally_reserved":0,"allocated":138723328,"data_stored":138547365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52756,"kb_used_data":3080,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3153920,"data_stored":2977957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244776,"kb_used_data":136800,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":94127064,"statfs":{"total":96636764160,"available":96386113536,"internally_reserved":0,"allocated":140083200,"data_stored":139907237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":110554831},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,8,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140091392,"data_stored":140091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3620864,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139038720,"data_stored":139038720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142319616,"data_stored":142319616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:07.695 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 46 20 35905 35885 6.09211 3 0.0592784 0.0256272 2024-04-26T02:25:08.695 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 47 20 36309 36289 6.02965 3.15625 0.0673098 0.0258932 2024-04-26T02:25:09.695 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 48 20 36710 36690 5.96932 3.13281 0.0525569 0.0261535 2024-04-26T02:25:10.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:10.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 49 20 37115 37095 5.91201 3.16406 0.0669279 0.0264114 2024-04-26T02:25:11.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 50 20 37484 37464 5.85144 2.88281 0.0408393 0.0266737 2024-04-26T02:25:11.773 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:11.774 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:11.937 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-26T02:25:10.438283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297622696,"num_objects":36337,"num_object_clones":0,"num_object_copies":145348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36364,"num_write_kb":290794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36391,"ondisk_log_size":36391,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":902256,"kb_used_data":444368,"kb_used_omap":14,"kb_used_meta":457841,"kb_avail":376585104,"statfs":{"total":386547056640,"available":385623146496,"internally_reserved":0,"allocated":455032832,"data_stored":454328980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":468829747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,18,8,9,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45072384,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":44016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.213146"},"pg_stats":[{"pgid":"1.f","version":"32'2251","reported_seq":2288,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.620792+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:09.620792+0000","last_peered":"2024-04-26T02:25:09.620792+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:09.620792+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2251,"num_write_kb":18008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2298","reported_seq":2359,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.156317+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:08.156317+0000","last_peered":"2024-04-26T02:25:08.156317+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:08.156317+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2298,"log_dups_size":0,"ondisk_log_size":2298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18788876,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":18364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2223","reported_seq":2296,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.464198+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:07.464198+0000","last_peered":"2024-04-26T02:25:07.464198+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:07.464198+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18184454,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2187","reported_seq":2245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.122582+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:08.122582+0000","last_peered":"2024-04-26T02:25:08.122582+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:08.122582+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2187,"log_dups_size":0,"ondisk_log_size":2187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17879564,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2185,"num_write_kb":17469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2264","reported_seq":2315,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.614432+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:09.614432+0000","last_peered":"2024-04-26T02:25:09.614432+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:09.614432+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18518540,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2289","reported_seq":2327,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.584029+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:09.584029+0000","last_peered":"2024-04-26T02:25:09.584029+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:09.584029+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2279","reported_seq":2604,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.244350+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:08.244350+0000","last_peered":"2024-04-26T02:25:08.244350+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:08.244350+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18616844,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":18196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2398","reported_seq":2459,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.175113+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:08.175113+0000","last_peered":"2024-04-26T02:25:08.175113+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:08.175113+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2398,"log_dups_size":0,"ondisk_log_size":2398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599884,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2396,"num_write_kb":19156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2272","reported_seq":2324,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.570211+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:09.570211+0000","last_peered":"2024-04-26T02:25:09.570211+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:09.570211+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18539544,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2269,"num_write_kb":18129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2237","reported_seq":2311,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.298283+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:07.298283+0000","last_peered":"2024-04-26T02:25:07.298283+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:07.298283+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2237,"log_dups_size":0,"ondisk_log_size":2237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18305548,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2235,"num_write_kb":17878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":198}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2243","reported_seq":2296,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.609191+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:09.609191+0000","last_peered":"2024-04-26T02:25:09.609191+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:09.609191+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18328338,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2240,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2239","reported_seq":2272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.604320+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:09.604320+0000","last_peered":"2024-04-26T02:25:09.604320+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:09.604320+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2239,"log_dups_size":0,"ondisk_log_size":2239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18331910,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2304","reported_seq":2374,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.463848+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:07.463848+0000","last_peered":"2024-04-26T02:25:07.463848+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:07.463848+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2304,"log_dups_size":0,"ondisk_log_size":2304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864390,"num_objects":2303,"num_object_clones":0,"num_object_copies":9212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2303,"num_write_kb":18423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2349","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.650183+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:09.650183+0000","last_peered":"2024-04-26T02:25:09.650183+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:09.650183+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2349,"log_dups_size":0,"ondisk_log_size":2349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19206668,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":274}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2257","reported_seq":2331,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.458262+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:07.458262+0000","last_peered":"2024-04-26T02:25:07.458262+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:07.458262+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2257,"log_dups_size":0,"ondisk_log_size":2257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461196,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2301","reported_seq":2361,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.217213+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:08.217213+0000","last_peered":"2024-04-26T02:25:08.217213+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:08.217213+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2301,"log_dups_size":0,"ondisk_log_size":2301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18805260,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2299,"num_write_kb":18380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":297622696,"num_objects":36337,"num_object_clones":0,"num_object_copies":145348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36364,"num_write_kb":290794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":456929280,"data_stored":456929280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36391,"ondisk_log_size":36391,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281512,"kb_used_data":145696,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94090328,"statfs":{"total":96636764160,"available":96348495872,"internally_reserved":0,"allocated":149192704,"data_stored":149016741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,8,2,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272692,"kb_used_data":142956,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94099148,"statfs":{"total":96636764160,"available":96357527552,"internally_reserved":0,"allocated":146386944,"data_stored":146210981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69208,"kb_used_data":11216,"kb_used_omap":4,"kb_used_meta":57979,"kb_avail":94302632,"statfs":{"total":96636764160,"available":96565895168,"internally_reserved":0,"allocated":11485184,"data_stored":11309221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":59371215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278844,"kb_used_data":144500,"kb_used_omap":4,"kb_used_meta":134331,"kb_avail":94092996,"statfs":{"total":96636764160,"available":96351227904,"internally_reserved":0,"allocated":147968000,"data_stored":147792037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":137555663},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,4,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147968000,"data_stored":147968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11776000,"data_stored":11776000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146845696,"data_stored":146845696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":150339584,"data_stored":150339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:11.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:12.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 51 20 37880 37860 5.79738 3.09375 0.0535126 0.0269323 2024-04-26T02:25:13.529 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:13.529 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:13.648 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-26T02:25:12.439103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297622696,"num_objects":36337,"num_object_clones":0,"num_object_copies":145348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36364,"num_write_kb":290794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36391,"ondisk_log_size":36391,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":897764,"kb_used_data":452356,"kb_used_omap":14,"kb_used_meta":445361,"kb_avail":376589596,"statfs":{"total":386547056640,"available":385627746304,"internally_reserved":0,"allocated":463212544,"data_stored":462508692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":456050227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,4,15,7,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36610048,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005239"},"pg_stats":[{"pgid":"1.f","version":"32'2251","reported_seq":2288,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.620792+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:09.620792+0000","last_peered":"2024-04-26T02:25:09.620792+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:09.620792+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2251,"num_write_kb":18008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2298","reported_seq":2359,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.156317+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:08.156317+0000","last_peered":"2024-04-26T02:25:08.156317+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:08.156317+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2298,"log_dups_size":0,"ondisk_log_size":2298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18788876,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":18364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2223","reported_seq":2296,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.464198+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:07.464198+0000","last_peered":"2024-04-26T02:25:07.464198+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:07.464198+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18184454,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":196}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2187","reported_seq":2245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.122582+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:08.122582+0000","last_peered":"2024-04-26T02:25:08.122582+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:08.122582+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2187,"log_dups_size":0,"ondisk_log_size":2187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17879564,"num_objects":2183,"num_object_clones":0,"num_object_copies":8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2185,"num_write_kb":17469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2264","reported_seq":2315,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.614432+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:09.614432+0000","last_peered":"2024-04-26T02:25:09.614432+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:09.614432+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18518540,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2289","reported_seq":2327,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.584029+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:09.584029+0000","last_peered":"2024-04-26T02:25:09.584029+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:09.584029+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2279","reported_seq":2604,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.244350+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:08.244350+0000","last_peered":"2024-04-26T02:25:08.244350+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:08.244350+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18616844,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":18196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2398","reported_seq":2459,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.175113+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:08.175113+0000","last_peered":"2024-04-26T02:25:08.175113+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:08.175113+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2398,"log_dups_size":0,"ondisk_log_size":2398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599884,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2396,"num_write_kb":19156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2272","reported_seq":2324,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.570211+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:09.570211+0000","last_peered":"2024-04-26T02:25:09.570211+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:09.570211+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18539544,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2269,"num_write_kb":18129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2237","reported_seq":2311,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.298283+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:07.298283+0000","last_peered":"2024-04-26T02:25:07.298283+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:07.298283+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2237,"log_dups_size":0,"ondisk_log_size":2237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18305548,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2235,"num_write_kb":17878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":198}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2243","reported_seq":2296,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.609191+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:09.609191+0000","last_peered":"2024-04-26T02:25:09.609191+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:09.609191+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18328338,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2240,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2239","reported_seq":2272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.604320+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:09.604320+0000","last_peered":"2024-04-26T02:25:09.604320+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:09.604320+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2239,"log_dups_size":0,"ondisk_log_size":2239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18331910,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2304","reported_seq":2374,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.463848+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:07.463848+0000","last_peered":"2024-04-26T02:25:07.463848+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:07.463848+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2304,"log_dups_size":0,"ondisk_log_size":2304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18864390,"num_objects":2303,"num_object_clones":0,"num_object_copies":9212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2303,"num_write_kb":18423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2349","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.650183+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:09.650183+0000","last_peered":"2024-04-26T02:25:09.650183+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:09.650183+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2349,"log_dups_size":0,"ondisk_log_size":2349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19206668,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":274}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2257","reported_seq":2331,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:07.458262+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:07.458262+0000","last_peered":"2024-04-26T02:25:07.458262+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:07.458262+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2257,"log_dups_size":0,"ondisk_log_size":2257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461196,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2301","reported_seq":2361,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:08.217213+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:08.217213+0000","last_peered":"2024-04-26T02:25:08.217213+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:08.217213+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2301,"log_dups_size":0,"ondisk_log_size":2301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18805260,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2299,"num_write_kb":18380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":297622696,"num_objects":36337,"num_object_clones":0,"num_object_copies":145348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36364,"num_write_kb":290794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":465321984,"data_stored":465321984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36391,"ondisk_log_size":36391,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281512,"kb_used_data":145696,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94090328,"statfs":{"total":96636764160,"available":96348495872,"internally_reserved":0,"allocated":149192704,"data_stored":149016741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,8,2,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272692,"kb_used_data":142956,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94099148,"statfs":{"total":96636764160,"available":96357527552,"internally_reserved":0,"allocated":146386944,"data_stored":146210981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64716,"kb_used_data":19204,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94307124,"statfs":{"total":96636764160,"available":96570494976,"internally_reserved":0,"allocated":19664896,"data_stored":19488933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":46591695},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278844,"kb_used_data":144500,"kb_used_omap":4,"kb_used_meta":134331,"kb_avail":94092996,"statfs":{"total":96636764160,"available":96351227904,"internally_reserved":0,"allocated":147968000,"data_stored":147792037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":137555663},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,4,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147968000,"data_stored":147968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20168704,"data_stored":20168704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146845696,"data_stored":146845696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":150339584,"data_stored":150339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2251", 'reported_seq': 2288, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.620792+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:09.620792+0000', 'last_peered': '2024-04-26T02:25:09.620792+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:09.620792+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2251, 'log_dups_size': 0, 'ondisk_log_size': 2251, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18440192, 'num_objects': 2251, 'num_object_clones': 0, 'num_object_copies': 9004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2251, 'num_write_kb': 18008, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2298", 'reported_seq': 2359, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:08.156317+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:08.156317+0000', 'last_peered': '2024-04-26T02:25:08.156317+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:08.156317+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2298, 'log_dups_size': 0, 'ondisk_log_size': 2298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18788876, 'num_objects': 2294, 'num_object_clones': 0, 'num_object_copies': 9176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2296, 'num_write_kb': 18364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2223", 'reported_seq': 2296, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:07.464198+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:07.464198+0000', 'last_peered': '2024-04-26T02:25:07.464198+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:07.464198+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2223, 'log_dups_size': 0, 'ondisk_log_size': 2223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18184454, 'num_objects': 2220, 'num_object_clones': 0, 'num_object_copies': 8880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2222, 'num_write_kb': 17766, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 196}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:13.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2187", 'reported_seq': 2245, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:08.122582+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:08.122582+0000', 'last_peered': '2024-04-26T02:25:08.122582+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:08.122582+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2187, 'log_dups_size': 0, 'ondisk_log_size': 2187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17879564, 'num_objects': 2183, 'num_object_clones': 0, 'num_object_copies': 8732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2185, 'num_write_kb': 17469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 215}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2264", 'reported_seq': 2315, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.614432+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:09.614432+0000', 'last_peered': '2024-04-26T02:25:09.614432+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:09.614432+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2264, 'log_dups_size': 0, 'ondisk_log_size': 2264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18518540, 'num_objects': 2261, 'num_object_clones': 0, 'num_object_copies': 9044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2262, 'num_write_kb': 18093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 250}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2289", 'reported_seq': 2327, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.584029+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:09.584029+0000', 'last_peered': '2024-04-26T02:25:09.584029+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:09.584029+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2289, 'log_dups_size': 0, 'ondisk_log_size': 2289, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18751488, 'num_objects': 2289, 'num_object_clones': 0, 'num_object_copies': 9156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2289, 'num_write_kb': 18312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 238}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2279", 'reported_seq': 2604, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:08.244350+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:08.244350+0000', 'last_peered': '2024-04-26T02:25:08.244350+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:08.244350+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2279, 'log_dups_size': 0, 'ondisk_log_size': 2279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18616844, 'num_objects': 2273, 'num_object_clones': 0, 'num_object_copies': 9092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2277, 'num_write_kb': 18196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 133, 'num_bytes_recovered': 1073152, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 322}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2398", 'reported_seq': 2459, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:08.175113+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:08.175113+0000', 'last_peered': '2024-04-26T02:25:08.175113+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:08.175113+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2398, 'log_dups_size': 0, 'ondisk_log_size': 2398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19599884, 'num_objects': 2393, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2396, 'num_write_kb': 19156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:13.650 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2272", 'reported_seq': 2324, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.570211+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:09.570211+0000', 'last_peered': '2024-04-26T02:25:09.570211+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:09.570211+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2272, 'log_dups_size': 0, 'ondisk_log_size': 2272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18539544, 'num_objects': 2264, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2269, 'num_write_kb': 18129, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 260}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2237", 'reported_seq': 2311, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:07.298283+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:07.298283+0000', 'last_peered': '2024-04-26T02:25:07.298283+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:07.298283+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2237, 'log_dups_size': 0, 'ondisk_log_size': 2237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18305548, 'num_objects': 2235, 'num_object_clones': 0, 'num_object_copies': 8940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2235, 'num_write_kb': 17878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 198}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2243", 'reported_seq': 2296, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.609191+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:09.609191+0000', 'last_peered': '2024-04-26T02:25:09.609191+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:09.609191+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2243, 'log_dups_size': 0, 'ondisk_log_size': 2243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18328338, 'num_objects': 2238, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2240, 'num_write_kb': 17908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 221}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2239", 'reported_seq': 2272, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.604320+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:09.604320+0000', 'last_peered': '2024-04-26T02:25:09.604320+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:09.604320+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2239, 'log_dups_size': 0, 'ondisk_log_size': 2239, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18331910, 'num_objects': 2238, 'num_object_clones': 0, 'num_object_copies': 8952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2238, 'num_write_kb': 17903, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2304", 'reported_seq': 2374, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:07.463848+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:07.463848+0000', 'last_peered': '2024-04-26T02:25:07.463848+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:07.463848+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2304, 'log_dups_size': 0, 'ondisk_log_size': 2304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18864390, 'num_objects': 2303, 'num_object_clones': 0, 'num_object_copies': 9212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2303, 'num_write_kb': 18423, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:13.651 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2349", 'reported_seq': 2397, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:09.650183+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:09.650183+0000', 'last_peered': '2024-04-26T02:25:09.650183+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:09.650183+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2349, 'log_dups_size': 0, 'ondisk_log_size': 2349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19206668, 'num_objects': 2345, 'num_object_clones': 0, 'num_object_copies': 9380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2347, 'num_write_kb': 18772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:13.652 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:13.652 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2257", 'reported_seq': 2331, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:07.458262+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:07.458262+0000', 'last_peered': '2024-04-26T02:25:07.458262+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:07.458262+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2257, 'log_dups_size': 0, 'ondisk_log_size': 2257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18461196, 'num_objects': 2254, 'num_object_clones': 0, 'num_object_copies': 9016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2255, 'num_write_kb': 18037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:13.652 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:13.652 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2301", 'reported_seq': 2361, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:08.217213+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:08.217213+0000', 'last_peered': '2024-04-26T02:25:08.217213+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:08.217213+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2301, 'log_dups_size': 0, 'ondisk_log_size': 2301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18805260, 'num_objects': 2296, 'num_object_clones': 0, 'num_object_copies': 9184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2299, 'num_write_kb': 18380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:13.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:13.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 52 20 38186 38166 5.73188 2.39062 0.0330026 0.0272389 2024-04-26T02:25:14.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 53 20 38581 38561 5.68197 3.08594 0.0265329 0.027483 2024-04-26T02:25:15.492 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:15.493 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32550/149628 objects degraded (21.754%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37407,"data_bytes":306388136,"bytes_used":950870016,"bytes_avail":385596186624,"bytes_total":386547056640,"degraded_objects":32550,"degraded_total":149628,"degraded_ratio":0.21753949795492822,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":129649,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"write_bytes_sec":3779633,"read_op_per_sec":0,"write_op_per_sec":461},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-26T02:25:08.437880+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:15.566 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:15.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:15.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 54 20 38950 38930 5.63014 2.88281 0.06088 0.0277327 2024-04-26T02:25:16.366 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:16.367 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:16.514 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-04-26T02:25:14.439584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306388136,"num_objects":37407,"num_object_clones":0,"num_object_copies":149628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37434,"num_write_kb":299354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37461,"ondisk_log_size":37461,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":928584,"kb_used_data":467880,"kb_used_omap":14,"kb_used_meta":460657,"kb_avail":376558776,"statfs":{"total":386547056640,"available":385596186624,"internally_reserved":0,"allocated":479109120,"data_stored":478405268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":471713331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,10,15,5,10,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45375488,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":44312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005262"},"pg_stats":[{"pgid":"1.f","version":"32'2251","reported_seq":2288,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.620792+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:09.620792+0000","last_peered":"2024-04-26T02:25:09.620792+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:09.620792+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18440192,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2251,"num_write_kb":18008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2433","reported_seq":2494,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:13.080870+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:13.080870+0000","last_peered":"2024-04-26T02:25:13.080870+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:13.080870+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2433,"log_dups_size":0,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19894796,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2327","reported_seq":2400,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:12.505641+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:12.505641+0000","last_peered":"2024-04-26T02:25:12.505641+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:12.505641+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19036422,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":18598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2299","reported_seq":2357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:13.080762+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:13.080762+0000","last_peered":"2024-04-26T02:25:13.080762+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:13.080762+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2299,"log_dups_size":0,"ondisk_log_size":2299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18797068,"num_objects":2295,"num_object_clones":0,"num_object_copies":9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2297,"num_write_kb":18365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2264","reported_seq":2315,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.614432+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:09.614432+0000","last_peered":"2024-04-26T02:25:09.614432+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:09.614432+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18518540,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":18093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2289","reported_seq":2327,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.584029+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:09.584029+0000","last_peered":"2024-04-26T02:25:09.584029+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:09.584029+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18751488,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2423","reported_seq":2865,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:13.211802+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:13.211802+0000","last_peered":"2024-04-26T02:25:13.211802+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:13.211802+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2423,"log_dups_size":0,"ondisk_log_size":2423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19796492,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2421,"num_write_kb":19348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2511","reported_seq":2572,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:13.113481+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:13.113481+0000","last_peered":"2024-04-26T02:25:13.113481+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:13.113481+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2511,"log_dups_size":0,"ondisk_log_size":2511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20525580,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":20060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2272","reported_seq":2324,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.570211+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:09.570211+0000","last_peered":"2024-04-26T02:25:09.570211+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:09.570211+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18539544,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2269,"num_write_kb":18129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2347","reported_seq":2421,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:12.459309+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:12.459309+0000","last_peered":"2024-04-26T02:25:12.459309+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:12.459309+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2347,"log_dups_size":0,"ondisk_log_size":2347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19206668,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2345,"num_write_kb":18758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":308}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2243","reported_seq":2296,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.609191+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:09.609191+0000","last_peered":"2024-04-26T02:25:09.609191+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:09.609191+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18328338,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2240,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2239","reported_seq":2272,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.604320+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:09.604320+0000","last_peered":"2024-04-26T02:25:09.604320+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:09.604320+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2239,"log_dups_size":0,"ondisk_log_size":2239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18331910,"num_objects":2238,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":250}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2425","reported_seq":2495,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:12.491048+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:12.491048+0000","last_peered":"2024-04-26T02:25:12.491048+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:12.491048+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2425,"log_dups_size":0,"ondisk_log_size":2425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19855622,"num_objects":2424,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":19391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":313}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2349","reported_seq":2397,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:09.650183+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:09.650183+0000","last_peered":"2024-04-26T02:25:09.650183+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:09.650183+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2349,"log_dups_size":0,"ondisk_log_size":2349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19206668,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":18772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":274}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2376","reported_seq":2450,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:12.550107+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:12.550107+0000","last_peered":"2024-04-26T02:25:12.550107+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:12.550107+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19436044,"num_objects":2373,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":18989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":320}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2413","reported_seq":2473,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:13.074005+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:13.074005+0000","last_peered":"2024-04-26T02:25:13.074005+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:13.074005+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2413,"log_dups_size":0,"ondisk_log_size":2413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19722764,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2411,"num_write_kb":19276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306388136,"num_objects":37407,"num_object_clones":0,"num_object_copies":149628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37434,"num_write_kb":299354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":481378304,"data_stored":481378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37461,"ondisk_log_size":37461,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281512,"kb_used_data":145696,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94090328,"statfs":{"total":96636764160,"available":96348495872,"internally_reserved":0,"allocated":149192704,"data_stored":149016741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,8,2,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":290216,"kb_used_data":150752,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94081624,"statfs":{"total":96636764160,"available":96339582976,"internally_reserved":0,"allocated":154370048,"data_stored":154194085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64716,"kb_used_data":19204,"kb_used_omap":4,"kb_used_meta":45499,"kb_avail":94307124,"statfs":{"total":96636764160,"available":96570494976,"internally_reserved":0,"allocated":19664896,"data_stored":19488933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":46591695},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292140,"kb_used_data":152228,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94079700,"statfs":{"total":96636764160,"available":96337612800,"internally_reserved":0,"allocated":155881472,"data_stored":155705509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,4,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155901952,"data_stored":155901952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20168704,"data_stored":20168704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154968064,"data_stored":154968064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":150339584,"data_stored":150339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:16.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 55 20 39272 39252 5.57352 2.51562 0.0626113 0.0280121 2024-04-26T02:25:17.696 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 56 20 39658 39638 5.52785 3.01562 0.0572749 0.0282433 2024-04-26T02:25:18.697 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 57 20 40034 40014 5.48239 2.9375 0.0147959 0.0284798 2024-04-26T02:25:19.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:19.697 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 58 20 40418 40398 5.4396 3 0.0616426 0.0287049 2024-04-26T02:25:20.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 59 20 40777 40757 5.39478 2.80469 0.0497202 0.0289295 2024-04-26T02:25:20.787 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:20.788 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:20.869 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-26T02:25:20.441139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":995636,"kb_used_data":505812,"kb_used_omap":14,"kb_used_meta":489777,"kb_avail":376491724,"statfs":{"total":386547056640,"available":385527525376,"internally_reserved":0,"allocated":517951488,"data_stored":517247636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":501532211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,8,13,14,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37478400,"num_objects":4575,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4575,"num_write_kb":36600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"1.f","version":"32'2482","reported_seq":2519,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.750622+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:19.750622+0000","last_peered":"2024-04-26T02:25:19.750622+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:19.750622+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2556","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.248378+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:18.248378+0000","last_peered":"2024-04-26T02:25:18.248378+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:18.248378+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2556,"log_dups_size":0,"ondisk_log_size":2556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20902412,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2554,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2437","reported_seq":2509,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.585780+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:17.585780+0000","last_peered":"2024-04-26T02:25:17.585780+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:17.585780+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19937542,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2406","reported_seq":2464,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.227500+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:18.227500+0000","last_peered":"2024-04-26T02:25:18.227500+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:18.227500+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673612,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2404,"num_write_kb":19221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2501","reported_seq":2552,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.755204+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:19.755204+0000","last_peered":"2024-04-26T02:25:19.755204+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:19.755204+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460044,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2525","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.792462+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:19.792462+0000","last_peered":"2024-04-26T02:25:19.792462+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:19.792462+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2525,"log_dups_size":0,"ondisk_log_size":2525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20684800,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":20200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2522","reported_seq":3124,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.285230+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:18.285230+0000","last_peered":"2024-04-26T02:25:18.285230+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:18.285230+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20607500,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2627","reported_seq":2688,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.289905+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:18.289905+0000","last_peered":"2024-04-26T02:25:18.289905+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:18.289905+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2627,"log_dups_size":0,"ondisk_log_size":2627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2499","reported_seq":2551,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.754613+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:19.754613+0000","last_peered":"2024-04-26T02:25:19.754613+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:19.754613+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20399128,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":19945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2442","reported_seq":2516,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.486481+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:17.486481+0000","last_peered":"2024-04-26T02:25:17.486481+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:17.486481+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2442,"log_dups_size":0,"ondisk_log_size":2442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19984908,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2440,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2475","reported_seq":2527,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.789839+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:19.789839+0000","last_peered":"2024-04-26T02:25:19.789839+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:19.789839+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2475,"log_dups_size":0,"ondisk_log_size":2475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20228882,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2508","reported_seq":2542,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.781884+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:19.781884+0000","last_peered":"2024-04-26T02:25:19.781884+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:19.781884+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535558,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2533","reported_seq":2603,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.349074+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:17.349074+0000","last_peered":"2024-04-26T02:25:17.349074+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:17.349074+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20740358,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2532,"num_write_kb":20255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2597","reported_seq":2645,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.767248+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:19.767248+0000","last_peered":"2024-04-26T02:25:19.767248+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:19.767248+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21238284,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":20756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2490","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.521259+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:17.521259+0000","last_peered":"2024-04-26T02:25:17.521259+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:17.521259+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2490,"log_dups_size":0,"ondisk_log_size":2490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20369932,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2488,"num_write_kb":19901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2533","reported_seq":2593,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.197055+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:18.197055+0000","last_peered":"2024-04-26T02:25:18.197055+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:18.197055+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705804,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":519217152,"data_stored":519217152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310284,"kb_used_data":161988,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94061556,"statfs":{"total":96636764160,"available":96319033344,"internally_reserved":0,"allocated":165875712,"data_stored":165699749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301512,"kb_used_data":157888,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94070328,"statfs":{"total":96636764160,"available":96328015872,"internally_reserved":0,"allocated":161677312,"data_stored":161501349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80528,"kb_used_data":26696,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94291312,"statfs":{"total":96636764160,"available":96554303488,"internally_reserved":0,"allocated":27336704,"data_stored":27160741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55111375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303312,"kb_used_data":159240,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068528,"statfs":{"total":96636764160,"available":96326172672,"internally_reserved":0,"allocated":163061760,"data_stored":162885797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,4,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163409920,"data_stored":163409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27791360,"data_stored":27791360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162263040,"data_stored":162263040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165752832,"data_stored":165752832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:20.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:21.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:25:21.701145+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.02916 2024-04-26T02:25:21.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:25:21.699 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 60 20 41132 41112 5.3511 2.77344 0.0189563 0.02916 2024-04-26T02:25:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:21.818 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:21.889 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-26T02:25:20.441139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":995636,"kb_used_data":505812,"kb_used_omap":14,"kb_used_meta":489777,"kb_avail":376491724,"statfs":{"total":386547056640,"available":385527525376,"internally_reserved":0,"allocated":517951488,"data_stored":517247636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":501532211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,8,13,14,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37478400,"num_objects":4575,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4575,"num_write_kb":36600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"1.f","version":"32'2482","reported_seq":2519,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.750622+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:19.750622+0000","last_peered":"2024-04-26T02:25:19.750622+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:19.750622+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2556","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.248378+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:18.248378+0000","last_peered":"2024-04-26T02:25:18.248378+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:18.248378+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2556,"log_dups_size":0,"ondisk_log_size":2556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20902412,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2554,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2437","reported_seq":2509,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.585780+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:17.585780+0000","last_peered":"2024-04-26T02:25:17.585780+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:17.585780+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19937542,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2406","reported_seq":2464,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.227500+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:18.227500+0000","last_peered":"2024-04-26T02:25:18.227500+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:18.227500+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673612,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2404,"num_write_kb":19221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2501","reported_seq":2552,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.755204+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:19.755204+0000","last_peered":"2024-04-26T02:25:19.755204+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:19.755204+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460044,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2525","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.792462+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:19.792462+0000","last_peered":"2024-04-26T02:25:19.792462+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:19.792462+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2525,"log_dups_size":0,"ondisk_log_size":2525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20684800,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":20200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2522","reported_seq":3124,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.285230+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:18.285230+0000","last_peered":"2024-04-26T02:25:18.285230+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:18.285230+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20607500,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2627","reported_seq":2688,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.289905+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:18.289905+0000","last_peered":"2024-04-26T02:25:18.289905+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:18.289905+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2627,"log_dups_size":0,"ondisk_log_size":2627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2499","reported_seq":2551,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.754613+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:19.754613+0000","last_peered":"2024-04-26T02:25:19.754613+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:19.754613+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20399128,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":19945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2442","reported_seq":2516,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.486481+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:17.486481+0000","last_peered":"2024-04-26T02:25:17.486481+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:17.486481+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2442,"log_dups_size":0,"ondisk_log_size":2442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19984908,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2440,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2475","reported_seq":2527,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.789839+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:19.789839+0000","last_peered":"2024-04-26T02:25:19.789839+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:19.789839+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2475,"log_dups_size":0,"ondisk_log_size":2475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20228882,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2508","reported_seq":2542,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.781884+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:19.781884+0000","last_peered":"2024-04-26T02:25:19.781884+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:19.781884+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535558,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2533","reported_seq":2603,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.349074+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:17.349074+0000","last_peered":"2024-04-26T02:25:17.349074+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:17.349074+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20740358,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2532,"num_write_kb":20255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2597","reported_seq":2645,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.767248+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:19.767248+0000","last_peered":"2024-04-26T02:25:19.767248+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:19.767248+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21238284,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":20756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2490","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.521259+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:17.521259+0000","last_peered":"2024-04-26T02:25:17.521259+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:17.521259+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2490,"log_dups_size":0,"ondisk_log_size":2490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20369932,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2488,"num_write_kb":19901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2533","reported_seq":2593,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.197055+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:18.197055+0000","last_peered":"2024-04-26T02:25:18.197055+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:18.197055+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705804,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":519217152,"data_stored":519217152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310284,"kb_used_data":161988,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94061556,"statfs":{"total":96636764160,"available":96319033344,"internally_reserved":0,"allocated":165875712,"data_stored":165699749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301512,"kb_used_data":157888,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94070328,"statfs":{"total":96636764160,"available":96328015872,"internally_reserved":0,"allocated":161677312,"data_stored":161501349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80528,"kb_used_data":26696,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94291312,"statfs":{"total":96636764160,"available":96554303488,"internally_reserved":0,"allocated":27336704,"data_stored":27160741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55111375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303312,"kb_used_data":159240,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068528,"statfs":{"total":96636764160,"available":96326172672,"internally_reserved":0,"allocated":163061760,"data_stored":162885797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,4,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163409920,"data_stored":163409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27791360,"data_stored":27791360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162263040,"data_stored":162263040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165752832,"data_stored":165752832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:21.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:21.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2482", 'reported_seq': 2519, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.750622+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:19.750622+0000', 'last_peered': '2024-04-26T02:25:19.750622+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:19.750622+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2482, 'log_dups_size': 0, 'ondisk_log_size': 2482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20332544, 'num_objects': 2482, 'num_object_clones': 0, 'num_object_copies': 9928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2482, 'num_write_kb': 19856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:21.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2556", 'reported_seq': 2617, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:18.248378+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:18.248378+0000', 'last_peered': '2024-04-26T02:25:18.248378+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:18.248378+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2556, 'log_dups_size': 0, 'ondisk_log_size': 2556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20902412, 'num_objects': 2552, 'num_object_clones': 0, 'num_object_copies': 10208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2554, 'num_write_kb': 20428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:21.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2437", 'reported_seq': 2509, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:17.585780+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:17.585780+0000', 'last_peered': '2024-04-26T02:25:17.585780+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:17.585780+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2437, 'log_dups_size': 0, 'ondisk_log_size': 2437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19937542, 'num_objects': 2434, 'num_object_clones': 0, 'num_object_copies': 9736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2436, 'num_write_kb': 19478, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 410}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2406", 'reported_seq': 2464, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:18.227500+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:18.227500+0000', 'last_peered': '2024-04-26T02:25:18.227500+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:18.227500+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2406, 'log_dups_size': 0, 'ondisk_log_size': 2406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19673612, 'num_objects': 2402, 'num_object_clones': 0, 'num_object_copies': 9608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2404, 'num_write_kb': 19221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2501", 'reported_seq': 2552, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.755204+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:19.755204+0000', 'last_peered': '2024-04-26T02:25:19.755204+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:19.755204+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2501, 'log_dups_size': 0, 'ondisk_log_size': 2501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20460044, 'num_objects': 2498, 'num_object_clones': 0, 'num_object_copies': 9992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2499, 'num_write_kb': 19989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 487}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2525", 'reported_seq': 2564, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.792462+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:19.792462+0000', 'last_peered': '2024-04-26T02:25:19.792462+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:19.792462+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2525, 'log_dups_size': 0, 'ondisk_log_size': 2525, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20684800, 'num_objects': 2525, 'num_object_clones': 0, 'num_object_copies': 10100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2525, 'num_write_kb': 20200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2522", 'reported_seq': 3124, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:18.285230+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:18.285230+0000', 'last_peered': '2024-04-26T02:25:18.285230+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:18.285230+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2522, 'log_dups_size': 0, 'ondisk_log_size': 2522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20607500, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2520, 'num_write_kb': 20140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 272, 'num_bytes_recovered': 2211840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 704}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:21.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2627", 'reported_seq': 2688, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:18.289905+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:18.289905+0000', 'last_peered': '2024-04-26T02:25:18.289905+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:18.289905+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2627, 'log_dups_size': 0, 'ondisk_log_size': 2627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21475852, 'num_objects': 2622, 'num_object_clones': 0, 'num_object_copies': 10488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2625, 'num_write_kb': 20988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2499", 'reported_seq': 2551, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.754613+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:19.754613+0000', 'last_peered': '2024-04-26T02:25:19.754613+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:19.754613+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2499, 'log_dups_size': 0, 'ondisk_log_size': 2499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20399128, 'num_objects': 2491, 'num_object_clones': 0, 'num_object_copies': 9964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2496, 'num_write_kb': 19945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 487}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2442", 'reported_seq': 2516, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:17.486481+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:17.486481+0000', 'last_peered': '2024-04-26T02:25:17.486481+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:17.486481+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2442, 'log_dups_size': 0, 'ondisk_log_size': 2442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19984908, 'num_objects': 2440, 'num_object_clones': 0, 'num_object_copies': 9760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2440, 'num_write_kb': 19518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 403}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2475", 'reported_seq': 2527, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.789839+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:19.789839+0000', 'last_peered': '2024-04-26T02:25:19.789839+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:19.789839+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2475, 'log_dups_size': 0, 'ondisk_log_size': 2475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20228882, 'num_objects': 2470, 'num_object_clones': 0, 'num_object_copies': 9880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2472, 'num_write_kb': 19764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 453}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2508", 'reported_seq': 2542, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.781884+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:19.781884+0000', 'last_peered': '2024-04-26T02:25:19.781884+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:19.781884+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2508, 'log_dups_size': 0, 'ondisk_log_size': 2508, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20535558, 'num_objects': 2507, 'num_object_clones': 0, 'num_object_copies': 10028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2507, 'num_write_kb': 20055, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2533", 'reported_seq': 2603, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:17.349074+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:17.349074+0000', 'last_peered': '2024-04-26T02:25:17.349074+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:17.349074+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2533, 'log_dups_size': 0, 'ondisk_log_size': 2533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20740358, 'num_objects': 2532, 'num_object_clones': 0, 'num_object_copies': 10128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2532, 'num_write_kb': 20255, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:21.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:21.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2597", 'reported_seq': 2645, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:19.767248+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:19.767248+0000', 'last_peered': '2024-04-26T02:25:19.767248+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:19.767248+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2597, 'log_dups_size': 0, 'ondisk_log_size': 2597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21238284, 'num_objects': 2593, 'num_object_clones': 0, 'num_object_copies': 10372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2595, 'num_write_kb': 20756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:21.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:21.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2490", 'reported_seq': 2564, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:17.521259+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:17.521259+0000', 'last_peered': '2024-04-26T02:25:17.521259+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:17.521259+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2490, 'log_dups_size': 0, 'ondisk_log_size': 2490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20369932, 'num_objects': 2487, 'num_object_clones': 0, 'num_object_copies': 9948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2488, 'num_write_kb': 19901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:21.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:21.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2533", 'reported_seq': 2593, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:18.197055+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:18.197055+0000', 'last_peered': '2024-04-26T02:25:18.197055+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:18.197055+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2533, 'log_dups_size': 0, 'ondisk_log_size': 2533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20705804, 'num_objects': 2528, 'num_object_clones': 0, 'num_object_copies': 10112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2531, 'num_write_kb': 20236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:21.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:22.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 61 20 41482 41462 5.30817 2.73438 0.0219271 0.0294167 2024-04-26T02:25:23.633 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:23.633 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32470/160316 objects degraded (20.254%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":120,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40079,"data_bytes":328277160,"bytes_used":1036091392,"bytes_avail":385510965248,"bytes_total":386547056640,"degraded_objects":32470,"degraded_total":160316,"degraded_ratio":0.20253748846029093,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94864,"recovering_keys_per_sec":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"write_bytes_sec":2553828,"read_op_per_sec":0,"write_op_per_sec":311},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T02:25:16.440230+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:23.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 62 19 41834 41815 5.26697 2.75781 0.0300513 0.0296345 2024-04-26T02:25:23.755 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:23.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:24.431 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:24.432 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:24.459 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-26T02:25:22.441620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1011808,"kb_used_data":513664,"kb_used_omap":14,"kb_used_meta":498097,"kb_avail":376475552,"statfs":{"total":386547056640,"available":385510965248,"internally_reserved":0,"allocated":525991936,"data_stored":525288084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":510051891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,8,14,16,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30654464,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":29936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003336"},"pg_stats":[{"pgid":"1.f","version":"32'2482","reported_seq":2519,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.750622+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:19.750622+0000","last_peered":"2024-04-26T02:25:19.750622+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:19.750622+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2556","reported_seq":2617,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.248378+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:18.248378+0000","last_peered":"2024-04-26T02:25:18.248378+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:18.248378+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2556,"log_dups_size":0,"ondisk_log_size":2556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20902412,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2554,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2437","reported_seq":2509,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.585780+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:17.585780+0000","last_peered":"2024-04-26T02:25:17.585780+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:17.585780+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19937542,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2406","reported_seq":2464,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.227500+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:18.227500+0000","last_peered":"2024-04-26T02:25:18.227500+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:18.227500+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673612,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2404,"num_write_kb":19221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2501","reported_seq":2552,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.755204+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:19.755204+0000","last_peered":"2024-04-26T02:25:19.755204+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:19.755204+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460044,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2525","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.792462+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:19.792462+0000","last_peered":"2024-04-26T02:25:19.792462+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:19.792462+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2525,"log_dups_size":0,"ondisk_log_size":2525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20684800,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":20200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2522","reported_seq":3124,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.285230+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:18.285230+0000","last_peered":"2024-04-26T02:25:18.285230+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:18.285230+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20607500,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":704}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2627","reported_seq":2688,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.289905+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:18.289905+0000","last_peered":"2024-04-26T02:25:18.289905+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:18.289905+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2627,"log_dups_size":0,"ondisk_log_size":2627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":20988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":481}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2499","reported_seq":2551,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.754613+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:19.754613+0000","last_peered":"2024-04-26T02:25:19.754613+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:19.754613+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20399128,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":19945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2442","reported_seq":2516,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.486481+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:17.486481+0000","last_peered":"2024-04-26T02:25:17.486481+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:17.486481+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2442,"log_dups_size":0,"ondisk_log_size":2442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19984908,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2440,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2475","reported_seq":2527,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.789839+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:19.789839+0000","last_peered":"2024-04-26T02:25:19.789839+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:19.789839+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2475,"log_dups_size":0,"ondisk_log_size":2475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20228882,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2508","reported_seq":2542,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.781884+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:19.781884+0000","last_peered":"2024-04-26T02:25:19.781884+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:19.781884+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535558,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":20055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2533","reported_seq":2603,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.349074+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:17.349074+0000","last_peered":"2024-04-26T02:25:17.349074+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:17.349074+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20740358,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2532,"num_write_kb":20255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2597","reported_seq":2645,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:19.767248+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:19.767248+0000","last_peered":"2024-04-26T02:25:19.767248+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:19.767248+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21238284,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":20756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":522}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2490","reported_seq":2564,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:17.521259+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:17.521259+0000","last_peered":"2024-04-26T02:25:17.521259+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:17.521259+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2490,"log_dups_size":0,"ondisk_log_size":2490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20369932,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2488,"num_write_kb":19901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2533","reported_seq":2593,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:18.197055+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:18.197055+0000","last_peered":"2024-04-26T02:25:18.197055+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:18.197055+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705804,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328277160,"num_objects":40079,"num_object_clones":0,"num_object_copies":160316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40106,"num_write_kb":320730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":527196160,"data_stored":527196160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310284,"kb_used_data":161988,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94061556,"statfs":{"total":96636764160,"available":96319033344,"internally_reserved":0,"allocated":165875712,"data_stored":165699749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301512,"kb_used_data":157888,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94070328,"statfs":{"total":96636764160,"available":96328015872,"internally_reserved":0,"allocated":161677312,"data_stored":161501349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96700,"kb_used_data":34548,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94275140,"statfs":{"total":96636764160,"available":96537743360,"internally_reserved":0,"allocated":35377152,"data_stored":35201189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63631055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303312,"kb_used_data":159240,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068528,"statfs":{"total":96636764160,"available":96326172672,"internally_reserved":0,"allocated":163061760,"data_stored":162885797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":147517135},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,4,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163409920,"data_stored":163409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35770368,"data_stored":35770368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162263040,"data_stored":162263040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165752832,"data_stored":165752832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:24.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 63 20 42102 42082 5.21649 2.08594 0.137461 0.0299336 2024-04-26T02:25:25.700 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 64 20 42472 42452 5.18015 2.89062 0.0768534 0.0301411 2024-04-26T02:25:26.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 65 20 42844 42824 5.14503 2.90625 0.0345047 0.0303468 2024-04-26T02:25:27.461 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:27.702 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 66 20 43220 43200 5.11159 2.9375 0.0309705 0.0305461 2024-04-26T02:25:28.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 67 20 43579 43559 5.07713 2.80469 0.0521027 0.0307476 2024-04-26T02:25:28.859 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:28.860 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:28.959 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-04-26T02:25:28.446989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350960808,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42875,"num_write_kb":342882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42902,"ondisk_log_size":42902,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1043652,"kb_used_data":556580,"kb_used_omap":14,"kb_used_meta":487025,"kb_avail":376443708,"statfs":{"total":386547056640,"available":385478356992,"internally_reserved":0,"allocated":569937920,"data_stored":569234068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":498714163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,10,11,6,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37617664,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006845"},"pg_stats":[{"pgid":"1.f","version":"32'2593","reported_seq":2630,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.861634+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:24.861634+0000","last_peered":"2024-04-26T02:25:24.861634+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:24.861634+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21241856,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":20744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2771","reported_seq":2832,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.365842+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:28.365842+0000","last_peered":"2024-04-26T02:25:28.365842+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:28.365842+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22663692,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2652","reported_seq":2725,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.603233+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:27.603233+0000","last_peered":"2024-04-26T02:25:27.603233+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:27.603233+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21698822,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2626","reported_seq":2684,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.307540+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:28.307540+0000","last_peered":"2024-04-26T02:25:28.307540+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:28.307540+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2599","reported_seq":2650,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.709847+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:24.709847+0000","last_peered":"2024-04-26T02:25:24.709847+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:24.709847+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2599,"log_dups_size":0,"ondisk_log_size":2599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21262860,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2624","reported_seq":2663,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.771401+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:24.771401+0000","last_peered":"2024-04-26T02:25:24.771401+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:24.771401+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2624,"log_dups_size":0,"ondisk_log_size":2624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2751","reported_seq":3712,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.274416+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:28.274416+0000","last_peered":"2024-04-26T02:25:28.274416+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:28.274416+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22483468,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2870","reported_seq":2931,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.243141+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:28.243141+0000","last_peered":"2024-04-26T02:25:28.243141+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:28.243141+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23466508,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2611","reported_seq":2660,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.861719+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:24.861719+0000","last_peered":"2024-04-26T02:25:24.861719+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:24.861719+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21316632,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2708","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.590368+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:27.590368+0000","last_peered":"2024-04-26T02:25:27.590368+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:27.590368+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163980,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2706,"num_write_kb":21646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":669}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2582","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.794939+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:24.794939+0000","last_peered":"2024-04-26T02:25:24.794939+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:24.794939+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2582,"log_dups_size":0,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21105426,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2601","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.801708+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:24.801708+0000","last_peered":"2024-04-26T02:25:24.801708+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:24.801708+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21297414,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2600,"num_write_kb":20799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2748","reported_seq":2818,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.571659+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:27.571659+0000","last_peered":"2024-04-26T02:25:27.571659+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:27.571659+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22501638,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2716","reported_seq":2764,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.843065+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:24.843065+0000","last_peered":"2024-04-26T02:25:24.843065+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:24.843065+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2716,"log_dups_size":0,"ondisk_log_size":2716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213132,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":21708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2695","reported_seq":2769,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.499229+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:27.499229+0000","last_peered":"2024-04-26T02:25:27.499229+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:27.499229+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22049292,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2755","reported_seq":2815,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.350532+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:28.350532+0000","last_peered":"2024-04-26T02:25:28.350532+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:28.350532+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22524428,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":22012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350960808,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42875,"num_write_kb":342882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570941440,"data_stored":570941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42902,"ondisk_log_size":42902,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308856,"kb_used_data":168880,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94062984,"statfs":{"total":96636764160,"available":96320495616,"internally_reserved":0,"allocated":172933120,"data_stored":172757157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,2,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311692,"kb_used_data":172228,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94060148,"statfs":{"total":96636764160,"available":96317591552,"internally_reserved":0,"allocated":176361472,"data_stored":176185509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109508,"kb_used_data":41788,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94262332,"statfs":{"total":96636764160,"available":96524627968,"internally_reserved":0,"allocated":42790912,"data_stored":42614949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":69332687},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":173684,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":177852416,"data_stored":177676453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178065408,"data_stored":178065408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43200512,"data_stored":43200512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176971776,"data_stored":176971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172703744,"data_stored":172703744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:28.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:29.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 68 20 43928 43908 5.04257 2.72656 0.0375504 0.0309693 2024-04-26T02:25:30.222 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:30.227 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:30.338 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-04-26T02:25:28.446989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350960808,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42875,"num_write_kb":342882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42902,"ondisk_log_size":42902,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1043652,"kb_used_data":556580,"kb_used_omap":14,"kb_used_meta":487025,"kb_avail":376443708,"statfs":{"total":386547056640,"available":385478356992,"internally_reserved":0,"allocated":569937920,"data_stored":569234068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":498714163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,10,11,6,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37617664,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4592,"num_write_kb":36736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006845"},"pg_stats":[{"pgid":"1.f","version":"32'2593","reported_seq":2630,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.861634+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:24.861634+0000","last_peered":"2024-04-26T02:25:24.861634+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:24.861634+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21241856,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":20744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2771","reported_seq":2832,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.365842+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:28.365842+0000","last_peered":"2024-04-26T02:25:28.365842+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:28.365842+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22663692,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2652","reported_seq":2725,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.603233+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:27.603233+0000","last_peered":"2024-04-26T02:25:27.603233+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:27.603233+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21698822,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2626","reported_seq":2684,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.307540+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:28.307540+0000","last_peered":"2024-04-26T02:25:28.307540+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:28.307540+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2599","reported_seq":2650,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.709847+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:24.709847+0000","last_peered":"2024-04-26T02:25:24.709847+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:24.709847+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2599,"log_dups_size":0,"ondisk_log_size":2599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21262860,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2624","reported_seq":2663,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.771401+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:24.771401+0000","last_peered":"2024-04-26T02:25:24.771401+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:24.771401+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2624,"log_dups_size":0,"ondisk_log_size":2624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2751","reported_seq":3712,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.274416+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:28.274416+0000","last_peered":"2024-04-26T02:25:28.274416+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:28.274416+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22483468,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2870","reported_seq":2931,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.243141+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:28.243141+0000","last_peered":"2024-04-26T02:25:28.243141+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:28.243141+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23466508,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2611","reported_seq":2660,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.861719+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:24.861719+0000","last_peered":"2024-04-26T02:25:24.861719+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:24.861719+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21316632,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2708","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.590368+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:27.590368+0000","last_peered":"2024-04-26T02:25:27.590368+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:27.590368+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163980,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2706,"num_write_kb":21646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":669}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2582","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.794939+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:24.794939+0000","last_peered":"2024-04-26T02:25:24.794939+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:24.794939+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2582,"log_dups_size":0,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21105426,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2601","reported_seq":2635,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.801708+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:24.801708+0000","last_peered":"2024-04-26T02:25:24.801708+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:24.801708+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21297414,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2600,"num_write_kb":20799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":612}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2748","reported_seq":2818,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.571659+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:27.571659+0000","last_peered":"2024-04-26T02:25:27.571659+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:27.571659+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22501638,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2716","reported_seq":2764,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:24.843065+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:24.843065+0000","last_peered":"2024-04-26T02:25:24.843065+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:24.843065+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2716,"log_dups_size":0,"ondisk_log_size":2716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22213132,"num_objects":2712,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":21708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2695","reported_seq":2769,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.499229+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:27.499229+0000","last_peered":"2024-04-26T02:25:27.499229+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:27.499229+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22049292,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2755","reported_seq":2815,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.350532+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:28.350532+0000","last_peered":"2024-04-26T02:25:28.350532+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:28.350532+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22524428,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":22012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350960808,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42875,"num_write_kb":342882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570941440,"data_stored":570941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42902,"ondisk_log_size":42902,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308856,"kb_used_data":168880,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94062984,"statfs":{"total":96636764160,"available":96320495616,"internally_reserved":0,"allocated":172933120,"data_stored":172757157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,2,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311692,"kb_used_data":172228,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94060148,"statfs":{"total":96636764160,"available":96317591552,"internally_reserved":0,"allocated":176361472,"data_stored":176185509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109508,"kb_used_data":41788,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94262332,"statfs":{"total":96636764160,"available":96524627968,"internally_reserved":0,"allocated":42790912,"data_stored":42614949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":69332687},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":173684,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":177852416,"data_stored":177676453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178065408,"data_stored":178065408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43200512,"data_stored":43200512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176971776,"data_stored":176971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172703744,"data_stored":172703744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:30.339 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:30.339 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2593", 'reported_seq': 2630, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.861634+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:24.861634+0000', 'last_peered': '2024-04-26T02:25:24.861634+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:24.861634+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2593, 'log_dups_size': 0, 'ondisk_log_size': 2593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21241856, 'num_objects': 2593, 'num_object_clones': 0, 'num_object_copies': 10372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2593, 'num_write_kb': 20744, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 576}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.339 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2771", 'reported_seq': 2832, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:28.365842+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:28.365842+0000', 'last_peered': '2024-04-26T02:25:28.365842+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:28.365842+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2771, 'log_dups_size': 0, 'ondisk_log_size': 2771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22663692, 'num_objects': 2767, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2769, 'num_write_kb': 22148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2652", 'reported_seq': 2725, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:27.603233+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:27.603233+0000', 'last_peered': '2024-04-26T02:25:27.603233+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:27.603233+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2652, 'log_dups_size': 0, 'ondisk_log_size': 2652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21698822, 'num_objects': 2649, 'num_object_clones': 0, 'num_object_copies': 10596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2651, 'num_write_kb': 21198, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 625}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2626", 'reported_seq': 2684, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:28.307540+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:28.307540+0000', 'last_peered': '2024-04-26T02:25:28.307540+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:28.307540+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2626, 'log_dups_size': 0, 'ondisk_log_size': 2626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21475852, 'num_objects': 2622, 'num_object_clones': 0, 'num_object_copies': 10488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2624, 'num_write_kb': 20981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2599", 'reported_seq': 2650, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.709847+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:24.709847+0000', 'last_peered': '2024-04-26T02:25:24.709847+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:24.709847+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2599, 'log_dups_size': 0, 'ondisk_log_size': 2599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21262860, 'num_objects': 2596, 'num_object_clones': 0, 'num_object_copies': 10384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2597, 'num_write_kb': 20773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 585}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:30.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2624", 'reported_seq': 2663, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.771401+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:24.771401+0000', 'last_peered': '2024-04-26T02:25:24.771401+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:24.771401+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2624, 'log_dups_size': 0, 'ondisk_log_size': 2624, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21495808, 'num_objects': 2624, 'num_object_clones': 0, 'num_object_copies': 10496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2624, 'num_write_kb': 20992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 573}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2751", 'reported_seq': 3712, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:28.274416+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:28.274416+0000', 'last_peered': '2024-04-26T02:25:28.274416+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:28.274416+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2751, 'log_dups_size': 0, 'ondisk_log_size': 2751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22483468, 'num_objects': 2745, 'num_object_clones': 0, 'num_object_copies': 10980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1632, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2749, 'num_write_kb': 21972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 452, 'num_bytes_recovered': 3686400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1113}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2870", 'reported_seq': 2931, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:28.243141+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:28.243141+0000', 'last_peered': '2024-04-26T02:25:28.243141+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:28.243141+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2870, 'log_dups_size': 0, 'ondisk_log_size': 2870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23466508, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2868, 'num_write_kb': 22932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 724}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2611", 'reported_seq': 2660, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.861719+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:24.861719+0000', 'last_peered': '2024-04-26T02:25:24.861719+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:24.861719+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2611, 'log_dups_size': 0, 'ondisk_log_size': 2611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21316632, 'num_objects': 2603, 'num_object_clones': 0, 'num_object_copies': 10412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2608, 'num_write_kb': 20841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2708", 'reported_seq': 2782, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:27.590368+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:27.590368+0000', 'last_peered': '2024-04-26T02:25:27.590368+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:27.590368+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2708, 'log_dups_size': 0, 'ondisk_log_size': 2708, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22163980, 'num_objects': 2706, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2706, 'num_write_kb': 21646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 669}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:30.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2582", 'reported_seq': 2635, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.794939+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:24.794939+0000', 'last_peered': '2024-04-26T02:25:24.794939+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:24.794939+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2582, 'log_dups_size': 0, 'ondisk_log_size': 2582, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21105426, 'num_objects': 2577, 'num_object_clones': 0, 'num_object_copies': 10308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2579, 'num_write_kb': 20620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2601", 'reported_seq': 2635, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.801708+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:24.801708+0000', 'last_peered': '2024-04-26T02:25:24.801708+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:24.801708+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2601, 'log_dups_size': 0, 'ondisk_log_size': 2601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21297414, 'num_objects': 2600, 'num_object_clones': 0, 'num_object_copies': 10400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2600, 'num_write_kb': 20799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2748", 'reported_seq': 2818, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:27.571659+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:27.571659+0000', 'last_peered': '2024-04-26T02:25:27.571659+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:27.571659+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2748, 'log_dups_size': 0, 'ondisk_log_size': 2748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22501638, 'num_objects': 2747, 'num_object_clones': 0, 'num_object_copies': 10988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2747, 'num_write_kb': 21975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2716", 'reported_seq': 2764, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:24.843065+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:24.843065+0000', 'last_peered': '2024-04-26T02:25:24.843065+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:24.843065+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2716, 'log_dups_size': 0, 'ondisk_log_size': 2716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22213132, 'num_objects': 2712, 'num_object_clones': 0, 'num_object_copies': 10848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2714, 'num_write_kb': 21708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2695", 'reported_seq': 2769, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:27.499229+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:27.499229+0000', 'last_peered': '2024-04-26T02:25:27.499229+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:27.499229+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2695, 'log_dups_size': 0, 'ondisk_log_size': 2695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22049292, 'num_objects': 2692, 'num_object_clones': 0, 'num_object_copies': 10768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2693, 'num_write_kb': 21541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:30.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:30.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2755", 'reported_seq': 2815, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:28.350532+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:28.350532+0000', 'last_peered': '2024-04-26T02:25:28.350532+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:28.350532+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2755, 'log_dups_size': 0, 'ondisk_log_size': 2755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22524428, 'num_objects': 2750, 'num_object_clones': 0, 'num_object_copies': 11000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2753, 'num_write_kb': 22012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:30.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:30.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 69 20 44294 44274 5.01094 2.85938 0.0612066 0.0311666 2024-04-26T02:25:31.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 70 20 44648 44628 4.97887 2.76562 0.0521758 0.0313564 2024-04-26T02:25:32.225 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:32.225 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32290/174604 objects degraded (18.493%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43651,"data_bytes":357538984,"bytes_used":1080373248,"bytes_avail":385466683392,"bytes_total":386547056640,"degraded_objects":32290,"degraded_total":174604,"degraded_ratio":0.18493276213603355,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":122794,"recovering_keys_per_sec":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":3003698,"read_op_per_sec":0,"write_op_per_sec":366},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T02:25:16.440230+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:32.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:32.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:32.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 71 20 45010 44990 4.94858 2.82812 0.0692635 0.0315546 2024-04-26T02:25:33.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 72 20 45363 45343 4.91816 2.75781 0.17754 0.0317422 2024-04-26T02:25:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:33.782 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:33.876 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-04-26T02:25:32.449629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357538984,"num_objects":43651,"num_object_clones":0,"num_object_copies":174604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43678,"num_write_kb":349306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43705,"ondisk_log_size":43705,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1088864,"kb_used_data":571260,"kb_used_omap":14,"kb_used_meta":517553,"kb_avail":376398496,"statfs":{"total":386547056640,"available":385432059904,"internally_reserved":0,"allocated":584970240,"data_stored":584266388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":529974835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,8,10,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29261824,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3572,"num_write_kb":28576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008489"},"pg_stats":[{"pgid":"1.f","version":"32'2696","reported_seq":2733,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.862586+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:29.862586+0000","last_peered":"2024-04-26T02:25:29.862586+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:29.862586+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2771","reported_seq":2832,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.365842+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:28.365842+0000","last_peered":"2024-04-26T02:25:28.365842+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:28.365842+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22663692,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2652","reported_seq":2725,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.603233+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:27.603233+0000","last_peered":"2024-04-26T02:25:27.603233+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:27.603233+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21698822,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2626","reported_seq":2684,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.307540+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:28.307540+0000","last_peered":"2024-04-26T02:25:28.307540+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:28.307540+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21475852,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2704","reported_seq":2755,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.867289+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:29.867289+0000","last_peered":"2024-04-26T02:25:29.867289+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:29.867289+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2704,"log_dups_size":0,"ondisk_log_size":2704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22123020,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2702,"num_write_kb":21613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":690}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2753","reported_seq":2792,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.829607+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:29.829607+0000","last_peered":"2024-04-26T02:25:29.829607+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:29.829607+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22552576,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":22024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2751","reported_seq":3712,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.274416+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:28.274416+0000","last_peered":"2024-04-26T02:25:28.274416+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:28.274416+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22483468,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2870","reported_seq":2931,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.243141+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:28.243141+0000","last_peered":"2024-04-26T02:25:28.243141+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:28.243141+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23466508,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2728","reported_seq":2780,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.813398+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:29.813398+0000","last_peered":"2024-04-26T02:25:29.813398+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:29.813398+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22275096,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":21777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2708","reported_seq":2782,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.590368+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:27.590368+0000","last_peered":"2024-04-26T02:25:27.590368+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:27.590368+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163980,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2706,"num_write_kb":21646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":669}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2693","reported_seq":2746,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.846335+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:29.846335+0000","last_peered":"2024-04-26T02:25:29.846335+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:29.846335+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2693,"log_dups_size":0,"ondisk_log_size":2693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22014738,"num_objects":2688,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2712","reported_seq":2745,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.838435+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:29.838435+0000","last_peered":"2024-04-26T02:25:29.838435+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:29.838435+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2712,"log_dups_size":0,"ondisk_log_size":2712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22206726,"num_objects":2711,"num_object_clones":0,"num_object_copies":10844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2711,"num_write_kb":21687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2748","reported_seq":2818,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.571659+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:27.571659+0000","last_peered":"2024-04-26T02:25:27.571659+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:27.571659+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22501638,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2843","reported_seq":2891,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:29.728380+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:29.728380+0000","last_peered":"2024-04-26T02:25:29.728380+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:29.728380+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23253516,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":768}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2695","reported_seq":2769,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:27.499229+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:27.499229+0000","last_peered":"2024-04-26T02:25:27.499229+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:27.499229+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22049292,"num_objects":2692,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2755","reported_seq":2815,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:28.350532+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:28.350532+0000","last_peered":"2024-04-26T02:25:28.350532+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:28.350532+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22524428,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":22012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":357538984,"num_objects":43651,"num_object_clones":0,"num_object_copies":174604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43678,"num_write_kb":349306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3686400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":586137600,"data_stored":586137600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43705,"ondisk_log_size":43705,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320256,"kb_used_data":176116,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94051584,"statfs":{"total":96636764160,"available":96308822016,"internally_reserved":0,"allocated":180342784,"data_stored":180166821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311692,"kb_used_data":172228,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94060148,"statfs":{"total":96636764160,"available":96317591552,"internally_reserved":0,"allocated":176361472,"data_stored":176185509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143320,"kb_used_data":49232,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94228520,"statfs":{"total":96636764160,"available":96490004480,"internally_reserved":0,"allocated":50413568,"data_stored":50237605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":173684,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":177852416,"data_stored":177676453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178065408,"data_stored":178065408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50900992,"data_stored":50900992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176971776,"data_stored":176971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180199424,"data_stored":180199424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:34.703 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 73 20 45739 45719 4.89103 2.9375 0.029372 0.0319319 2024-04-26T02:25:35.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 74 20 46108 46088 4.8639 2.88281 0.0178592 0.0320963 2024-04-26T02:25:36.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 75 20 46460 46440 4.83571 2.75 0.0396162 0.0322992 2024-04-26T02:25:36.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:37.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 76 20 46814 46794 4.80848 2.76562 0.0743267 0.0324755 2024-04-26T02:25:38.455 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:38.455 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:38.517 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-04-26T02:25:36.450413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372153512,"num_objects":45435,"num_object_clones":0,"num_object_copies":181740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45462,"num_write_kb":363578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45489,"ondisk_log_size":45489,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1134152,"kb_used_data":591592,"kb_used_omap":14,"kb_used_meta":542513,"kb_avail":376353208,"statfs":{"total":386547056640,"available":385385684992,"internally_reserved":0,"allocated":605790208,"data_stored":605086356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":555533875},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,13,11,9,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35364864,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":34536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008364"},"pg_stats":[{"pgid":"1.f","version":"32'2803","reported_seq":2840,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.822904+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:34.822904+0000","last_peered":"2024-04-26T02:25:34.822904+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:34.822904+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22962176,"num_objects":2803,"num_object_clones":0,"num_object_copies":11212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":22424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2872","reported_seq":2933,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.360414+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:33.360414+0000","last_peered":"2024-04-26T02:25:33.360414+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:33.360414+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23491084,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2753","reported_seq":2825,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:32.654614+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:32.654614+0000","last_peered":"2024-04-26T02:25:32.654614+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:32.654614+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22526214,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":22006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":726}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2732","reported_seq":2790,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.353929+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:33.353929+0000","last_peered":"2024-04-26T02:25:33.353929+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:33.353929+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22344204,"num_objects":2728,"num_object_clones":0,"num_object_copies":10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2814","reported_seq":2864,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.882682+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:34.882682+0000","last_peered":"2024-04-26T02:25:34.882682+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:34.882682+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23024140,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2873","reported_seq":2912,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.868612+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:34.868612+0000","last_peered":"2024-04-26T02:25:34.868612+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:34.868612+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2873,"log_dups_size":0,"ondisk_log_size":2873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23535616,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":22984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2874","reported_seq":3962,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.446478+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:33.446478+0000","last_peered":"2024-04-26T02:25:33.446478+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:33.446478+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2874,"log_dups_size":0,"ondisk_log_size":2874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23491084,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2872,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2977","reported_seq":3038,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.512580+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:33.512580+0000","last_peered":"2024-04-26T02:25:33.512580+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:33.512580+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2977,"log_dups_size":0,"ondisk_log_size":2977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24343052,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2840","reported_seq":2892,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.844635+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:34.844635+0000","last_peered":"2024-04-26T02:25:34.844635+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:34.844635+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23192600,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2814","reported_seq":2888,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:32.533893+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:32.533893+0000","last_peered":"2024-04-26T02:25:32.533893+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:32.533893+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23032332,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":22494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2791","reported_seq":2844,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.795647+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:34.795647+0000","last_peered":"2024-04-26T02:25:34.795647+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:34.795647+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22817554,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":22292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2840","reported_seq":2874,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.889364+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:34.889364+0000","last_peered":"2024-04-26T02:25:34.889364+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:34.889364+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23255302,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2839,"num_write_kb":22711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":851}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2855","reported_seq":2925,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:32.593997+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:32.593997+0000","last_peered":"2024-04-26T02:25:32.593997+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:32.593997+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2855,"log_dups_size":0,"ondisk_log_size":2855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23378182,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":22831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":743}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2970","reported_seq":3018,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.854226+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:34.854226+0000","last_peered":"2024-04-26T02:25:34.854226+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:34.854226+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24293900,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2800","reported_seq":2874,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:32.632640+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:32.632640+0000","last_peered":"2024-04-26T02:25:32.632640+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:32.632640+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2800,"log_dups_size":0,"ondisk_log_size":2800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22909452,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":22381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":744}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2881","reported_seq":2941,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.473879+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:33.473879+0000","last_peered":"2024-04-26T02:25:33.473879+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:33.473879+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23556620,"num_objects":2876,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":812}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372153512,"num_objects":45435,"num_object_clones":0,"num_object_copies":181740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45462,"num_write_kb":363578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":608399360,"data_stored":608399360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45489,"ondisk_log_size":45489,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334540,"kb_used_data":182084,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94037300,"statfs":{"total":96636764160,"available":96294195200,"internally_reserved":0,"allocated":186454016,"data_stored":186278053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":156102352},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327204,"kb_used_data":179420,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94044636,"statfs":{"total":96636764160,"available":96301707264,"internally_reserved":0,"allocated":183726080,"data_stored":183550117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143320,"kb_used_data":49232,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94228520,"statfs":{"total":96636764160,"available":96490004480,"internally_reserved":0,"allocated":50413568,"data_stored":50237605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329088,"kb_used_data":180856,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94042752,"statfs":{"total":96636764160,"available":96299778048,"internally_reserved":0,"allocated":185196544,"data_stored":185020581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185552896,"data_stored":185552896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50900992,"data_stored":50900992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184279040,"data_stored":184279040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187666432,"data_stored":187666432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:38.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:38.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 77 20 47165 47145 4.78165 2.74219 0.0790565 0.0326573 2024-04-26T02:25:39.704 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 78 20 47504 47484 4.7543 2.64844 0.0605654 0.0328472 2024-04-26T02:25:39.889 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:39.890 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:40.034 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-04-26T02:25:38.450873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376118440,"num_objects":45919,"num_object_clones":0,"num_object_copies":183676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45946,"num_write_kb":367450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45973,"ondisk_log_size":45973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1141984,"kb_used_data":606336,"kb_used_omap":14,"kb_used_meta":535601,"kb_avail":376345376,"statfs":{"total":386547056640,"available":385377665024,"internally_reserved":0,"allocated":620888064,"data_stored":620184212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":548455987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,12,14,13,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33275904,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008352"},"pg_stats":[{"pgid":"1.f","version":"32'2803","reported_seq":2840,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.822904+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:34.822904+0000","last_peered":"2024-04-26T02:25:34.822904+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:34.822904+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22962176,"num_objects":2803,"num_object_clones":0,"num_object_copies":11212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":22424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2872","reported_seq":2933,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.360414+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:33.360414+0000","last_peered":"2024-04-26T02:25:33.360414+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:33.360414+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23491084,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2868","reported_seq":2941,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.574678+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:37.574678+0000","last_peered":"2024-04-26T02:25:37.574678+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:37.574678+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2868,"log_dups_size":0,"ondisk_log_size":2868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468294,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":22926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2732","reported_seq":2790,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.353929+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:33.353929+0000","last_peered":"2024-04-26T02:25:33.353929+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:33.353929+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22344204,"num_objects":2728,"num_object_clones":0,"num_object_copies":10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2814","reported_seq":2864,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.882682+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:34.882682+0000","last_peered":"2024-04-26T02:25:34.882682+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:34.882682+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23024140,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":22493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2873","reported_seq":2912,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.868612+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:34.868612+0000","last_peered":"2024-04-26T02:25:34.868612+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:34.868612+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2873,"log_dups_size":0,"ondisk_log_size":2873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23535616,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":22984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2874","reported_seq":3962,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.446478+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:33.446478+0000","last_peered":"2024-04-26T02:25:33.446478+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:33.446478+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2874,"log_dups_size":0,"ondisk_log_size":2874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23491084,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2872,"num_write_kb":22956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2977","reported_seq":3038,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.512580+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:33.512580+0000","last_peered":"2024-04-26T02:25:33.512580+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:33.512580+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2977,"log_dups_size":0,"ondisk_log_size":2977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24343052,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2840","reported_seq":2892,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.844635+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:34.844635+0000","last_peered":"2024-04-26T02:25:34.844635+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:34.844635+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23192600,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2949","reported_seq":3020,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.689701+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:37.689701+0000","last_peered":"2024-04-26T02:25:37.689701+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:37.689701+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138252,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2791","reported_seq":2844,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.795647+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:34.795647+0000","last_peered":"2024-04-26T02:25:34.795647+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:34.795647+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22817554,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":22292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2840","reported_seq":2874,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.889364+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:34.889364+0000","last_peered":"2024-04-26T02:25:34.889364+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:34.889364+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23255302,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2839,"num_write_kb":22711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":851}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2974","reported_seq":3044,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.521268+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:37.521268+0000","last_peered":"2024-04-26T02:25:37.521268+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:37.521268+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24353030,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'2970","reported_seq":3018,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:34.854226+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:34.854226+0000","last_peered":"2024-04-26T02:25:34.854226+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:34.854226+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24293900,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":23740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2915","reported_seq":2986,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.614526+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:37.614526+0000","last_peered":"2024-04-26T02:25:37.614526+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:37.614526+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23851532,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":23301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'2881","reported_seq":2941,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:33.473879+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:33.473879+0000","last_peered":"2024-04-26T02:25:33.473879+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:33.473879+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23556620,"num_objects":2876,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":812}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376118440,"num_objects":45919,"num_object_clones":0,"num_object_copies":183676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45946,"num_write_kb":367450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":4202496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":623558656,"data_stored":623558656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45973,"ondisk_log_size":45973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334540,"kb_used_data":182084,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94037300,"statfs":{"total":96636764160,"available":96294195200,"internally_reserved":0,"allocated":186454016,"data_stored":186278053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":156102352},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340012,"kb_used_data":186660,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94031828,"statfs":{"total":96636764160,"available":96288591872,"internally_reserved":0,"allocated":191139840,"data_stored":190963877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138344,"kb_used_data":56736,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233496,"statfs":{"total":96636764160,"available":96495099904,"internally_reserved":0,"allocated":58097664,"data_stored":57921701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329088,"kb_used_data":180856,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94042752,"statfs":{"total":96636764160,"available":96299778048,"internally_reserved":0,"allocated":185196544,"data_stored":185020581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185552896,"data_stored":185552896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58601472,"data_stored":58601472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191737856,"data_stored":191737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187666432,"data_stored":187666432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2803", 'reported_seq': 2840, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.822904+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:34.822904+0000', 'last_peered': '2024-04-26T02:25:34.822904+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:34.822904+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2803, 'log_dups_size': 0, 'ondisk_log_size': 2803, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22962176, 'num_objects': 2803, 'num_object_clones': 0, 'num_object_copies': 11212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2803, 'num_write_kb': 22424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'2872", 'reported_seq': 2933, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:33.360414+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:33.360414+0000', 'last_peered': '2024-04-26T02:25:33.360414+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:33.360414+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2872, 'log_dups_size': 0, 'ondisk_log_size': 2872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23491084, 'num_objects': 2868, 'num_object_clones': 0, 'num_object_copies': 11472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2870, 'num_write_kb': 22956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2868", 'reported_seq': 2941, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:37.574678+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:37.574678+0000', 'last_peered': '2024-04-26T02:25:37.574678+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:37.574678+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2868, 'log_dups_size': 0, 'ondisk_log_size': 2868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23468294, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2867, 'num_write_kb': 22926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 841}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:40.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2732", 'reported_seq': 2790, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:33.353929+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:33.353929+0000', 'last_peered': '2024-04-26T02:25:33.353929+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:33.353929+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2732, 'log_dups_size': 0, 'ondisk_log_size': 2732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22344204, 'num_objects': 2728, 'num_object_clones': 0, 'num_object_copies': 10912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2730, 'num_write_kb': 21829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'2814", 'reported_seq': 2864, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.882682+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:34.882682+0000', 'last_peered': '2024-04-26T02:25:34.882682+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:34.882682+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2814, 'log_dups_size': 0, 'ondisk_log_size': 2814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23024140, 'num_objects': 2811, 'num_object_clones': 0, 'num_object_copies': 11244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2812, 'num_write_kb': 22493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 800}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'2873", 'reported_seq': 2912, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.868612+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:34.868612+0000', 'last_peered': '2024-04-26T02:25:34.868612+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:34.868612+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2873, 'log_dups_size': 0, 'ondisk_log_size': 2873, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23535616, 'num_objects': 2873, 'num_object_clones': 0, 'num_object_copies': 11492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2873, 'num_write_kb': 22984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'2874", 'reported_seq': 3962, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:33.446478+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:33.446478+0000', 'last_peered': '2024-04-26T02:25:33.446478+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:33.446478+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2874, 'log_dups_size': 0, 'ondisk_log_size': 2874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23491084, 'num_objects': 2868, 'num_object_clones': 0, 'num_object_copies': 11472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1569, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2872, 'num_write_kb': 22956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 515, 'num_bytes_recovered': 4202496, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'2977", 'reported_seq': 3038, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:33.512580+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:33.512580+0000', 'last_peered': '2024-04-26T02:25:33.512580+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:33.512580+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2977, 'log_dups_size': 0, 'ondisk_log_size': 2977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24343052, 'num_objects': 2972, 'num_object_clones': 0, 'num_object_copies': 11888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2975, 'num_write_kb': 23788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:40.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'2840", 'reported_seq': 2892, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.844635+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:34.844635+0000', 'last_peered': '2024-04-26T02:25:34.844635+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:34.844635+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2840, 'log_dups_size': 0, 'ondisk_log_size': 2840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23192600, 'num_objects': 2832, 'num_object_clones': 0, 'num_object_copies': 11328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2837, 'num_write_kb': 22673, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 828}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'2949", 'reported_seq': 3020, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:37.689701+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:37.689701+0000', 'last_peered': '2024-04-26T02:25:37.689701+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:37.689701+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2949, 'log_dups_size': 0, 'ondisk_log_size': 2949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24138252, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2947, 'num_write_kb': 23574, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 910}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'2791", 'reported_seq': 2844, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.795647+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:34.795647+0000', 'last_peered': '2024-04-26T02:25:34.795647+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:34.795647+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2791, 'log_dups_size': 0, 'ondisk_log_size': 2791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22817554, 'num_objects': 2786, 'num_object_clones': 0, 'num_object_copies': 11144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2788, 'num_write_kb': 22292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 769}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'2840", 'reported_seq': 2874, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.889364+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:34.889364+0000', 'last_peered': '2024-04-26T02:25:34.889364+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:34.889364+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2840, 'log_dups_size': 0, 'ondisk_log_size': 2840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23255302, 'num_objects': 2839, 'num_object_clones': 0, 'num_object_copies': 11356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2839, 'num_write_kb': 22711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'2974", 'reported_seq': 3044, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:37.521268+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:37.521268+0000', 'last_peered': '2024-04-26T02:25:37.521268+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:37.521268+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2974, 'log_dups_size': 0, 'ondisk_log_size': 2974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24353030, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2973, 'num_write_kb': 23783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 862}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'2970", 'reported_seq': 3018, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:34.854226+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:34.854226+0000', 'last_peered': '2024-04-26T02:25:34.854226+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:34.854226+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2970, 'log_dups_size': 0, 'ondisk_log_size': 2970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24293900, 'num_objects': 2966, 'num_object_clones': 0, 'num_object_copies': 11864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2968, 'num_write_kb': 23740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 895}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:40.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'2915", 'reported_seq': 2986, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:37.614526+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:37.614526+0000', 'last_peered': '2024-04-26T02:25:37.614526+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:37.614526+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2915, 'log_dups_size': 0, 'ondisk_log_size': 2915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23851532, 'num_objects': 2912, 'num_object_clones': 0, 'num_object_copies': 11648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2913, 'num_write_kb': 23301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:40.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:40.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'2881", 'reported_seq': 2941, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:33.473879+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:33.473879+0000', 'last_peered': '2024-04-26T02:25:33.473879+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:33.473879+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2881, 'log_dups_size': 0, 'ondisk_log_size': 2881, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23556620, 'num_objects': 2876, 'num_object_clones': 0, 'num_object_copies': 11504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2879, 'num_write_kb': 23020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:40.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:40.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 79 20 47844 47824 4.72775 2.65625 0.0567104 0.0330271 2024-04-26T02:25:41.039 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:41.039 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32150/189080 objects degraded (17.003%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":137,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47270,"data_bytes":387185832,"bytes_used":1195569152,"bytes_avail":385351487488,"bytes_total":386547056640,"degraded_objects":32150,"degraded_total":189080,"degraded_ratio":0.17003384810662156,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":95538,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":3017666,"read_op_per_sec":0,"write_op_per_sec":368},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T02:25:16.440230+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:41.078 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:41.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:41.703 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:41.704 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:41.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:25:41.708728+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0332538 2024-04-26T02:25:41.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:25:41.706 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 80 20 48123 48103 4.69578 2.17969 0.0319993 0.0332538 2024-04-26T02:25:41.770 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-04-26T02:25:40.451307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387185832,"num_objects":47270,"num_object_clones":0,"num_object_copies":189080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47297,"num_write_kb":378258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47324,"ondisk_log_size":47324,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1167548,"kb_used_data":620764,"kb_used_omap":14,"kb_used_meta":546737,"kb_avail":376319812,"statfs":{"total":386547056640,"available":385351487488,"internally_reserved":0,"allocated":635662336,"data_stored":634958484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":559859251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,14,10,15,14,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36225024,"num_objects":4422,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004317"},"pg_stats":[{"pgid":"1.f","version":"32'2893","reported_seq":2929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.923137+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:39.923137+0000","last_peered":"2024-04-26T02:25:39.923137+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:39.923137+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23699456,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2988","reported_seq":3049,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:38.636179+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:38.636179+0000","last_peered":"2024-04-26T02:25:38.636179+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:38.636179+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2988,"log_dups_size":0,"ondisk_log_size":2988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24441356,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2986,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2868","reported_seq":2941,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.574678+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:37.574678+0000","last_peered":"2024-04-26T02:25:37.574678+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:37.574678+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2868,"log_dups_size":0,"ondisk_log_size":2868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23468294,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":22926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2845","reported_seq":2901,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:38.605682+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:38.605682+0000","last_peered":"2024-04-26T02:25:38.605682+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:38.605682+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2845,"log_dups_size":0,"ondisk_log_size":2845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23269900,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2843,"num_write_kb":22733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":873}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2926","reported_seq":2977,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.702404+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:39.702404+0000","last_peered":"2024-04-26T02:25:39.702404+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:39.702404+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23941644,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":912}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2987","reported_seq":3026,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.883747+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:39.883747+0000","last_peered":"2024-04-26T02:25:39.883747+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:39.883747+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24469504,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'2977","reported_seq":4220,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:38.618211+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:38.618211+0000","last_peered":"2024-04-26T02:25:38.618211+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:38.618211+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2977,"log_dups_size":0,"ondisk_log_size":2977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24334860,"num_objects":2971,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1479}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3092","reported_seq":3153,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:38.592590+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:38.592590+0000","last_peered":"2024-04-26T02:25:38.592590+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:38.592590+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3092,"log_dups_size":0,"ondisk_log_size":3092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285132,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3090,"num_write_kb":24708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2950","reported_seq":3002,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.886675+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:39.886675+0000","last_peered":"2024-04-26T02:25:39.886675+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:39.886675+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24093720,"num_objects":2942,"num_object_clones":0,"num_object_copies":11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2949","reported_seq":3020,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.689701+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:37.689701+0000","last_peered":"2024-04-26T02:25:37.689701+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:37.689701+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24138252,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2923","reported_seq":2976,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.847193+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:39.847193+0000","last_peered":"2024-04-26T02:25:39.847193+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:39.847193+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23898898,"num_objects":2918,"num_object_clones":0,"num_object_copies":11672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":23348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2945","reported_seq":2979,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.767321+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:39.767321+0000","last_peered":"2024-04-26T02:25:39.767321+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:39.767321+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24115462,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":23551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2974","reported_seq":3044,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.521268+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:37.521268+0000","last_peered":"2024-04-26T02:25:37.521268+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:37.521268+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24353030,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3082","reported_seq":3128,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.907342+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:39.907342+0000","last_peered":"2024-04-26T02:25:39.907342+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:39.907342+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25211404,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3080,"num_write_kb":24636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'2915","reported_seq":2986,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:37.614526+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:37.614526+0000","last_peered":"2024-04-26T02:25:37.614526+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:37.614526+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23851532,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":23301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3010","reported_seq":3070,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:38.561612+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:38.561612+0000","last_peered":"2024-04-26T02:25:38.561612+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:38.561612+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3010,"log_dups_size":0,"ondisk_log_size":3010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24613388,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3008,"num_write_kb":24052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":387185832,"num_objects":47270,"num_object_clones":0,"num_object_copies":189080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47297,"num_write_kb":378258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638349312,"data_stored":638349312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47324,"ondisk_log_size":47324,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347336,"kb_used_data":189312,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94024504,"statfs":{"total":96636764160,"available":96281092096,"internally_reserved":0,"allocated":193855488,"data_stored":193679525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":161803984},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,8,4,8,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340012,"kb_used_data":186660,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94031828,"statfs":{"total":96636764160,"available":96288591872,"internally_reserved":0,"allocated":191139840,"data_stored":190963877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138344,"kb_used_data":56736,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94233496,"statfs":{"total":96636764160,"available":96495099904,"internally_reserved":0,"allocated":58097664,"data_stored":57921701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341856,"kb_used_data":188056,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94029984,"statfs":{"total":96636764160,"available":96286703616,"internally_reserved":0,"allocated":192569344,"data_stored":192393381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193118208,"data_stored":193118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58601472,"data_stored":58601472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":191737856,"data_stored":191737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194891776,"data_stored":194891776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:42.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 81 20 48480 48460 4.67224 2.78906 0.0223861 0.0334229 2024-04-26T02:25:43.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 82 20 48838 48818 4.64938 2.79688 0.0362269 0.0335847 2024-04-26T02:25:44.707 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 83 20 49193 49173 4.62678 2.77344 0.0947031 0.033749 2024-04-26T02:25:44.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:45.578 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:45.579 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:45.653 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-26T02:25:44.454088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394894504,"num_objects":48211,"num_object_clones":0,"num_object_copies":192844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48238,"num_write_kb":385786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48265,"ondisk_log_size":48265,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1244992,"kb_used_data":641312,"kb_used_omap":14,"kb_used_meta":603633,"kb_avail":376242368,"statfs":{"total":386547056640,"available":385272184832,"internally_reserved":0,"allocated":656703488,"data_stored":655999636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":618120755},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,7,10,7,14,10,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37355520,"num_objects":4560,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4560,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1884160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004459"},"pg_stats":[{"pgid":"1.f","version":"32'2893","reported_seq":2929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.923137+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:39.923137+0000","last_peered":"2024-04-26T02:25:39.923137+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:39.923137+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23699456,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3105","reported_seq":3166,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.709534+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:43.709534+0000","last_peered":"2024-04-26T02:25:43.709534+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:43.709534+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25399820,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":24820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2968","reported_seq":3041,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.692718+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:42.692718+0000","last_peered":"2024-04-26T02:25:42.692718+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:42.692718+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287494,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2967,"num_write_kb":23726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":941}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2929","reported_seq":2986,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.677649+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:43.677649+0000","last_peered":"2024-04-26T02:25:43.677649+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:43.677649+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958028,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'2926","reported_seq":2977,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.702404+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:39.702404+0000","last_peered":"2024-04-26T02:25:39.702404+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:39.702404+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23941644,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":23389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":912}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'2987","reported_seq":3026,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.883747+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:39.883747+0000","last_peered":"2024-04-26T02:25:39.883747+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:39.883747+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24469504,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3081","reported_seq":4504,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.680026+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:43.680026+0000","last_peered":"2024-04-26T02:25:43.680026+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:43.680026+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25186828,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3198","reported_seq":3258,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.685408+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:43.685408+0000","last_peered":"2024-04-26T02:25:43.685408+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:43.685408+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26153484,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2950","reported_seq":3002,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.886675+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:39.886675+0000","last_peered":"2024-04-26T02:25:39.886675+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:39.886675+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24093720,"num_objects":2942,"num_object_clones":0,"num_object_copies":11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":23553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3045","reported_seq":3119,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.591431+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:42.591431+0000","last_peered":"2024-04-26T02:25:42.591431+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:42.591431+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3045,"log_dups_size":0,"ondisk_log_size":3045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24924684,"num_objects":3043,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":24342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1006}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'2923","reported_seq":2976,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.847193+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:39.847193+0000","last_peered":"2024-04-26T02:25:39.847193+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:39.847193+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23898898,"num_objects":2918,"num_object_clones":0,"num_object_copies":11672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":23348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":901}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'2945","reported_seq":2979,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.767321+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:39.767321+0000","last_peered":"2024-04-26T02:25:39.767321+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:39.767321+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24115462,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":23551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3089","reported_seq":3159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.686639+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:42.686639+0000","last_peered":"2024-04-26T02:25:42.686639+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:42.686639+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25295110,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":24703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":977}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3082","reported_seq":3128,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:39.907342+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:39.907342+0000","last_peered":"2024-04-26T02:25:39.907342+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:39.907342+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25211404,"num_objects":3078,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3080,"num_write_kb":24636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3026","reported_seq":3100,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.618868+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:42.618868+0000","last_peered":"2024-04-26T02:25:42.618868+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:42.618868+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24760844,"num_objects":3023,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3024,"num_write_kb":24189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3118","reported_seq":3178,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.687385+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:43.687385+0000","last_peered":"2024-04-26T02:25:43.687385+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:43.687385+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498124,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":394894504,"num_objects":48211,"num_object_clones":0,"num_object_copies":192844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48238,"num_write_kb":385786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":659329024,"data_stored":659329024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48265,"ondisk_log_size":48265,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347336,"kb_used_data":189312,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94024504,"statfs":{"total":96636764160,"available":96281092096,"internally_reserved":0,"allocated":193855488,"data_stored":193679525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":161803984},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,8,4,8,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368864,"kb_used_data":193304,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94002976,"statfs":{"total":96636764160,"available":96259047424,"internally_reserved":0,"allocated":197943296,"data_stored":197767333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153824,"kb_used_data":63896,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94218016,"statfs":{"total":96636764160,"available":96479248384,"internally_reserved":0,"allocated":65429504,"data_stored":65253541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374968,"kb_used_data":194800,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93996872,"statfs":{"total":96636764160,"available":96252796928,"internally_reserved":0,"allocated":199475200,"data_stored":199299237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200048640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65830912,"data_stored":65830912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198557696,"data_stored":198557696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194891776,"data_stored":194891776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:45.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:45.708 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 84 20 49491 49471 4.59935 2.32812 0.0306708 0.0339519 2024-04-26T02:25:46.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 85 20 49824 49804 4.57584 2.60156 0.0475251 0.0341268 2024-04-26T02:25:46.973 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:46.973 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:47.130 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-04-26T02:25:46.454547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400981160,"num_objects":48954,"num_object_clones":0,"num_object_copies":195816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48981,"num_write_kb":391730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49008,"ondisk_log_size":49008,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1278124,"kb_used_data":648076,"kb_used_omap":14,"kb_used_meta":630001,"kb_avail":376209236,"statfs":{"total":386547056640,"available":385238257664,"internally_reserved":0,"allocated":663629824,"data_stored":662925972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":645121587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,4,9,6,8,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35397632,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004508"},"pg_stats":[{"pgid":"1.f","version":"32'2983","reported_seq":3020,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.894316+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:44.894316+0000","last_peered":"2024-04-26T02:25:44.894316+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:44.894316+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24436736,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3105","reported_seq":3166,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.709534+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:43.709534+0000","last_peered":"2024-04-26T02:25:43.709534+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:43.709534+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25399820,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":24820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'2968","reported_seq":3041,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.692718+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:42.692718+0000","last_peered":"2024-04-26T02:25:42.692718+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:42.692718+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2968,"log_dups_size":0,"ondisk_log_size":2968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287494,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2967,"num_write_kb":23726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":941}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2929","reported_seq":2986,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.677649+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:43.677649+0000","last_peered":"2024-04-26T02:25:43.677649+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:43.677649+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958028,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3028","reported_seq":3079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.782643+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:44.782643+0000","last_peered":"2024-04-26T02:25:44.782643+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:44.782643+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3028,"log_dups_size":0,"ondisk_log_size":3028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777228,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3105","reported_seq":3144,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.949257+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:44.949257+0000","last_peered":"2024-04-26T02:25:44.949257+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:44.949257+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25436160,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3081","reported_seq":4504,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.680026+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:43.680026+0000","last_peered":"2024-04-26T02:25:43.680026+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:43.680026+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25186828,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3198","reported_seq":3258,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.685408+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:43.685408+0000","last_peered":"2024-04-26T02:25:43.685408+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:43.685408+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26153484,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3059","reported_seq":3111,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.810332+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:44.810332+0000","last_peered":"2024-04-26T02:25:44.810332+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:44.810332+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3059,"log_dups_size":0,"ondisk_log_size":3059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24986648,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3045","reported_seq":3119,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.591431+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:42.591431+0000","last_peered":"2024-04-26T02:25:42.591431+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:42.591431+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3045,"log_dups_size":0,"ondisk_log_size":3045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24924684,"num_objects":3043,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":24342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1006}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3032","reported_seq":3085,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:45.036642+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:45.036642+0000","last_peered":"2024-04-26T02:25:45.036642+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:45.036642+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24791826,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3045","reported_seq":3079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.889471+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:44.889471+0000","last_peered":"2024-04-26T02:25:44.889471+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:44.889471+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3045,"log_dups_size":0,"ondisk_log_size":3045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24934662,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3044,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3089","reported_seq":3159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.686639+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:42.686639+0000","last_peered":"2024-04-26T02:25:42.686639+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:42.686639+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25295110,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":24703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":977}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3197","reported_seq":3245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.868824+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:44.868824+0000","last_peered":"2024-04-26T02:25:44.868824+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:44.868824+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26153484,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3026","reported_seq":3100,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:42.618868+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:42.618868+0000","last_peered":"2024-04-26T02:25:42.618868+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:42.618868+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24760844,"num_objects":3023,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3024,"num_write_kb":24189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3118","reported_seq":3178,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.687385+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:43.687385+0000","last_peered":"2024-04-26T02:25:43.687385+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:43.687385+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498124,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":400981160,"num_objects":48954,"num_object_clones":0,"num_object_copies":195816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48981,"num_write_kb":391730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":666361856,"data_stored":666361856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49008,"ondisk_log_size":49008,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380468,"kb_used_data":196076,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93991372,"statfs":{"total":96636764160,"available":96247164928,"internally_reserved":0,"allocated":200781824,"data_stored":200605861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,6,0,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368864,"kb_used_data":193304,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94002976,"statfs":{"total":96636764160,"available":96259047424,"internally_reserved":0,"allocated":197943296,"data_stored":197767333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153824,"kb_used_data":63896,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94218016,"statfs":{"total":96636764160,"available":96479248384,"internally_reserved":0,"allocated":65429504,"data_stored":65253541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374968,"kb_used_data":194800,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93996872,"statfs":{"total":96636764160,"available":96252796928,"internally_reserved":0,"allocated":199475200,"data_stored":199299237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200048640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65830912,"data_stored":65830912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198557696,"data_stored":198557696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201924608,"data_stored":201924608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:47.131 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'2983", 'reported_seq': 3020, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.894316+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:44.894316+0000', 'last_peered': '2024-04-26T02:25:44.894316+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:44.894316+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2983, 'log_dups_size': 0, 'ondisk_log_size': 2983, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24436736, 'num_objects': 2983, 'num_object_clones': 0, 'num_object_copies': 11932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2983, 'num_write_kb': 23864, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3105", 'reported_seq': 3166, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:43.709534+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:43.709534+0000', 'last_peered': '2024-04-26T02:25:43.709534+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:43.709534+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3105, 'log_dups_size': 0, 'ondisk_log_size': 3105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25399820, 'num_objects': 3101, 'num_object_clones': 0, 'num_object_copies': 12404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3103, 'num_write_kb': 24820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'2968", 'reported_seq': 3041, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:42.692718+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:42.692718+0000', 'last_peered': '2024-04-26T02:25:42.692718+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:42.692718+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2968, 'log_dups_size': 0, 'ondisk_log_size': 2968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24287494, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2967, 'num_write_kb': 23726, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 941}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'2929", 'reported_seq': 2986, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:43.677649+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:43.677649+0000', 'last_peered': '2024-04-26T02:25:43.677649+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:43.677649+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 2929, 'log_dups_size': 0, 'ondisk_log_size': 2929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23958028, 'num_objects': 2925, 'num_object_clones': 0, 'num_object_copies': 11700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2927, 'num_write_kb': 23405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:47.132 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3028", 'reported_seq': 3079, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.782643+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:44.782643+0000', 'last_peered': '2024-04-26T02:25:44.782643+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:44.782643+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3028, 'log_dups_size': 0, 'ondisk_log_size': 3028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24777228, 'num_objects': 3025, 'num_object_clones': 0, 'num_object_copies': 12100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3026, 'num_write_kb': 24205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1014}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3105", 'reported_seq': 3144, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.949257+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:44.949257+0000', 'last_peered': '2024-04-26T02:25:44.949257+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:44.949257+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3105, 'log_dups_size': 0, 'ondisk_log_size': 3105, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25436160, 'num_objects': 3105, 'num_object_clones': 0, 'num_object_copies': 12420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3105, 'num_write_kb': 24840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3081", 'reported_seq': 4504, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:43.680026+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:43.680026+0000', 'last_peered': '2024-04-26T02:25:43.680026+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:43.680026+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3081, 'log_dups_size': 0, 'ondisk_log_size': 3081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25186828, 'num_objects': 3075, 'num_object_clones': 0, 'num_object_copies': 12300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1402, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3079, 'num_write_kb': 24612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 682, 'num_bytes_recovered': 5570560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1673}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3198", 'reported_seq': 3258, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:43.685408+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:43.685408+0000', 'last_peered': '2024-04-26T02:25:43.685408+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:43.685408+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3198, 'log_dups_size': 0, 'ondisk_log_size': 3198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26153484, 'num_objects': 3193, 'num_object_clones': 0, 'num_object_copies': 12772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3196, 'num_write_kb': 25556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3059", 'reported_seq': 3111, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.810332+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:44.810332+0000', 'last_peered': '2024-04-26T02:25:44.810332+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:44.810332+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3059, 'log_dups_size': 0, 'ondisk_log_size': 3059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24986648, 'num_objects': 3051, 'num_object_clones': 0, 'num_object_copies': 12204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3056, 'num_write_kb': 24425, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:47.133 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3045", 'reported_seq': 3119, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:42.591431+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:42.591431+0000', 'last_peered': '2024-04-26T02:25:42.591431+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:42.591431+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3045, 'log_dups_size': 0, 'ondisk_log_size': 3045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24924684, 'num_objects': 3043, 'num_object_clones': 0, 'num_object_copies': 12172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3043, 'num_write_kb': 24342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1006}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3032", 'reported_seq': 3085, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:45.036642+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:45.036642+0000', 'last_peered': '2024-04-26T02:25:45.036642+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:45.036642+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3032, 'log_dups_size': 0, 'ondisk_log_size': 3032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24791826, 'num_objects': 3027, 'num_object_clones': 0, 'num_object_copies': 12108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3029, 'num_write_kb': 24220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1010}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3045", 'reported_seq': 3079, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.889471+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:44.889471+0000', 'last_peered': '2024-04-26T02:25:44.889471+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:44.889471+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3045, 'log_dups_size': 0, 'ondisk_log_size': 3045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24934662, 'num_objects': 3044, 'num_object_clones': 0, 'num_object_copies': 12176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3044, 'num_write_kb': 24351, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1056}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3089", 'reported_seq': 3159, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:42.686639+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:42.686639+0000', 'last_peered': '2024-04-26T02:25:42.686639+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:42.686639+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3089, 'log_dups_size': 0, 'ondisk_log_size': 3089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25295110, 'num_objects': 3088, 'num_object_clones': 0, 'num_object_copies': 12352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3088, 'num_write_kb': 24703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3197", 'reported_seq': 3245, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:44.868824+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:44.868824+0000', 'last_peered': '2024-04-26T02:25:44.868824+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:44.868824+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3197, 'log_dups_size': 0, 'ondisk_log_size': 3197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26153484, 'num_objects': 3193, 'num_object_clones': 0, 'num_object_copies': 12772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3195, 'num_write_kb': 25556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:47.134 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3026", 'reported_seq': 3100, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:42.618868+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:42.618868+0000', 'last_peered': '2024-04-26T02:25:42.618868+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:42.618868+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3026, 'log_dups_size': 0, 'ondisk_log_size': 3026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24760844, 'num_objects': 3023, 'num_object_clones': 0, 'num_object_copies': 12092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3024, 'num_write_kb': 24189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:47.135 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:47.135 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3118", 'reported_seq': 3178, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:43.687385+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:43.687385+0000', 'last_peered': '2024-04-26T02:25:43.687385+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:43.687385+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3118, 'log_dups_size': 0, 'ondisk_log_size': 3118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25498124, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3116, 'num_write_kb': 24916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:47.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:47.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 86 20 50159 50139 4.55307 2.61719 0.194902 0.0342926 2024-04-26T02:25:48.690 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:48.690 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32060/197460 objects degraded (16.236%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":145,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49365,"data_bytes":404348072,"bytes_used":1335472128,"bytes_avail":385211584512,"bytes_total":386547056640,"degraded_objects":32060,"degraded_total":197460,"degraded_ratio":0.16236199736655524,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":113961,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"write_bytes_sec":2681845,"read_op_per_sec":0,"write_op_per_sec":327},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T02:25:16.440230+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:48.709 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 87 20 50494 50474 4.53078 2.61719 0.125477 0.0344676 2024-04-26T02:25:48.802 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:48.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:49.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 88 20 50838 50818 4.50978 2.6875 0.0204194 0.0346299 2024-04-26T02:25:50.074 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:50.075 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:50.193 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-04-26T02:25:48.455041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404348072,"num_objects":49365,"num_object_clones":0,"num_object_copies":197460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49392,"num_write_kb":395018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49419,"ondisk_log_size":49419,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1304172,"kb_used_data":661644,"kb_used_omap":14,"kb_used_meta":642481,"kb_avail":376183188,"statfs":{"total":386547056640,"available":385211584512,"internally_reserved":0,"allocated":677523456,"data_stored":676819604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":657901107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,8,10,9,9,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32194560,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":31440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004627"},"pg_stats":[{"pgid":"1.f","version":"32'2983","reported_seq":3020,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.894316+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:44.894316+0000","last_peered":"2024-04-26T02:25:44.894316+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:44.894316+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2983,"log_dups_size":0,"ondisk_log_size":2983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24436736,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3105","reported_seq":3166,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.709534+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:43.709534+0000","last_peered":"2024-04-26T02:25:43.709534+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:43.709534+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25399820,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":24820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1025}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3056","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.647209+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:47.647209+0000","last_peered":"2024-04-26T02:25:47.647209+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:47.647209+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25008390,"num_objects":3053,"num_object_clones":0,"num_object_copies":12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3055,"num_write_kb":24430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'2929","reported_seq":2986,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.677649+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:43.677649+0000","last_peered":"2024-04-26T02:25:43.677649+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:43.677649+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958028,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":23405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3028","reported_seq":3079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.782643+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:44.782643+0000","last_peered":"2024-04-26T02:25:44.782643+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:44.782643+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3028,"log_dups_size":0,"ondisk_log_size":3028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777228,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3105","reported_seq":3144,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.949257+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:44.949257+0000","last_peered":"2024-04-26T02:25:44.949257+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:44.949257+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25436160,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3081","reported_seq":4504,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.680026+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:43.680026+0000","last_peered":"2024-04-26T02:25:43.680026+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:43.680026+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25186828,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3198","reported_seq":3258,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.685408+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:43.685408+0000","last_peered":"2024-04-26T02:25:43.685408+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:43.685408+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26153484,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3059","reported_seq":3111,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.810332+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:44.810332+0000","last_peered":"2024-04-26T02:25:44.810332+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:44.810332+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3059,"log_dups_size":0,"ondisk_log_size":3059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24986648,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3139","reported_seq":3214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.752330+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:47.752330+0000","last_peered":"2024-04-26T02:25:47.752330+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:47.752330+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25694732,"num_objects":3137,"num_object_clones":0,"num_object_copies":12548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3032","reported_seq":3085,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:45.036642+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:45.036642+0000","last_peered":"2024-04-26T02:25:45.036642+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:45.036642+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24791826,"num_objects":3027,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3045","reported_seq":3079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.889471+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:44.889471+0000","last_peered":"2024-04-26T02:25:44.889471+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:44.889471+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3045,"log_dups_size":0,"ondisk_log_size":3045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24934662,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3044,"num_write_kb":24351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3191","reported_seq":3261,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.666915+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:47.666915+0000","last_peered":"2024-04-26T02:25:47.666915+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:47.666915+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26130694,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":25519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3197","reported_seq":3245,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:44.868824+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:44.868824+0000","last_peered":"2024-04-26T02:25:44.868824+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:44.868824+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26153484,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3153","reported_seq":3227,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.739318+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:47.739318+0000","last_peered":"2024-04-26T02:25:47.739318+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:47.739318+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3153,"log_dups_size":0,"ondisk_log_size":3153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25801228,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3151,"num_write_kb":25205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3118","reported_seq":3178,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:43.687385+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:43.687385+0000","last_peered":"2024-04-26T02:25:43.687385+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:43.687385+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25498124,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":24916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404348072,"num_objects":49365,"num_object_clones":0,"num_object_copies":197460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49392,"num_write_kb":395018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5570560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":680529920,"data_stored":680529920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49419,"ondisk_log_size":49419,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380468,"kb_used_data":196076,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93991372,"statfs":{"total":96636764160,"available":96247164928,"internally_reserved":0,"allocated":200781824,"data_stored":200605861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,6,0,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383876,"kb_used_data":199996,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93987964,"statfs":{"total":96636764160,"available":96243675136,"internally_reserved":0,"allocated":204795904,"data_stored":204619941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,1,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164860,"kb_used_data":70772,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94206980,"statfs":{"total":96636764160,"available":96467947520,"internally_reserved":0,"allocated":72470528,"data_stored":72294565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374968,"kb_used_data":194800,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93996872,"statfs":{"total":96636764160,"available":96252796928,"internally_reserved":0,"allocated":199475200,"data_stored":199299237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200048640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73084928,"data_stored":73084928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205471744,"data_stored":205471744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201924608,"data_stored":201924608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:50.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 89 20 51187 51167 4.48975 2.72656 0.0762598 0.0347855 2024-04-26T02:25:51.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 90 20 51545 51525 4.47094 2.79688 0.0504153 0.0349319 2024-04-26T02:25:52.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 91 20 51897 51877 4.45203 2.75 0.0160934 0.0350779 2024-04-26T02:25:53.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:53.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 92 20 52193 52173 4.42878 2.3125 0.0104678 0.0352627 2024-04-26T02:25:54.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:54.045 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:54.115 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-26T02:25:52.457275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":414825640,"num_objects":50644,"num_object_clones":0,"num_object_copies":202576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50671,"num_write_kb":405250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50698,"ondisk_log_size":50698,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1303908,"kb_used_data":682180,"kb_used_omap":14,"kb_used_meta":621681,"kb_avail":376183452,"statfs":{"total":386547056640,"available":385211854848,"internally_reserved":0,"allocated":698552320,"data_stored":697848468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":636601907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,4,2,8,15,15,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27639808,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3374,"num_write_kb":26992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":1318912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005967"},"pg_stats":[{"pgid":"1.f","version":"32'3083","reported_seq":3120,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.116678+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:50.116678+0000","last_peered":"2024-04-26T02:25:50.116678+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:50.116678+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25255936,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3225","reported_seq":3286,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:48.734806+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:48.734806+0000","last_peered":"2024-04-26T02:25:48.734806+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:48.734806+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26382860,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3223,"num_write_kb":25780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3056","reported_seq":3129,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.647209+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:47.647209+0000","last_peered":"2024-04-26T02:25:47.647209+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:47.647209+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25008390,"num_objects":3053,"num_object_clones":0,"num_object_copies":12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3055,"num_write_kb":24430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3025","reported_seq":3083,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:48.742968+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:48.742968+0000","last_peered":"2024-04-26T02:25:48.742968+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:48.742968+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24744460,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3023,"num_write_kb":24173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3115","reported_seq":3166,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:49.966788+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:49.966788+0000","last_peered":"2024-04-26T02:25:49.966788+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:49.966788+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25489932,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3219","reported_seq":3258,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.056656+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:50.056656+0000","last_peered":"2024-04-26T02:25:50.056656+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:50.056656+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3219,"log_dups_size":0,"ondisk_log_size":3219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26370048,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3189","reported_seq":4754,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:48.762644+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:48.762644+0000","last_peered":"2024-04-26T02:25:48.762644+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:48.762644+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26071564,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":25476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1852}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3304","reported_seq":3365,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:48.735216+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:48.735216+0000","last_peered":"2024-04-26T02:25:48.735216+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:48.735216+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3304,"log_dups_size":0,"ondisk_log_size":3304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021836,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":26404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3160","reported_seq":3212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:49.924892+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:49.924892+0000","last_peered":"2024-04-26T02:25:49.924892+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:49.924892+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3160,"log_dups_size":0,"ondisk_log_size":3160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25814040,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3139","reported_seq":3214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.752330+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:47.752330+0000","last_peered":"2024-04-26T02:25:47.752330+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:47.752330+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25694732,"num_objects":3137,"num_object_clones":0,"num_object_copies":12548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":25094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3146","reported_seq":3199,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.098191+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:50.098191+0000","last_peered":"2024-04-26T02:25:50.098191+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:50.098191+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3146,"log_dups_size":0,"ondisk_log_size":3146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25725714,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3143,"num_write_kb":25132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3161","reported_seq":3195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.096796+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:50.096796+0000","last_peered":"2024-04-26T02:25:50.096796+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:50.096796+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3161,"log_dups_size":0,"ondisk_log_size":3161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25884934,"num_objects":3160,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":25279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1172}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3191","reported_seq":3261,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.666915+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:47.666915+0000","last_peered":"2024-04-26T02:25:47.666915+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:47.666915+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26130694,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":25519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3309","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.032955+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:50.032955+0000","last_peered":"2024-04-26T02:25:50.032955+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:50.032955+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27070988,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1234}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3153","reported_seq":3227,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:47.739318+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:47.739318+0000","last_peered":"2024-04-26T02:25:47.739318+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:47.739318+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3153,"log_dups_size":0,"ondisk_log_size":3153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25801228,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3151,"num_write_kb":25205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3223","reported_seq":3283,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:48.699979+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:48.699979+0000","last_peered":"2024-04-26T02:25:48.699979+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:48.699979+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3223,"log_dups_size":0,"ondisk_log_size":3223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26358284,"num_objects":3218,"num_object_clones":0,"num_object_copies":12872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":25756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":414825640,"num_objects":50644,"num_object_clones":0,"num_object_copies":202576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50671,"num_write_kb":405250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6152192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":701702144,"data_stored":701702144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50698,"ondisk_log_size":50698,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374812,"kb_used_data":202900,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93997028,"statfs":{"total":96636764160,"available":96252956672,"internally_reserved":0,"allocated":207769600,"data_stored":207593637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,2,6,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383876,"kb_used_data":199996,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93987964,"statfs":{"total":96636764160,"available":96243675136,"internally_reserved":0,"allocated":204795904,"data_stored":204619941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,1,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159348,"kb_used_data":77740,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94212492,"statfs":{"total":96636764160,"available":96473591808,"internally_reserved":0,"allocated":79605760,"data_stored":79429797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385872,"kb_used_data":201544,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93985968,"statfs":{"total":96636764160,"available":96241631232,"internally_reserved":0,"allocated":206381056,"data_stored":206205093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206913536,"data_stored":206913536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80465920,"data_stored":80465920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205471744,"data_stored":205471744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208850944,"data_stored":208850944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:54.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:54.711 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 93 20 52522 52502 4.4088 2.57031 0.0498842 0.0354202 2024-04-26T02:25:55.401 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:55.402 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:55.429 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-04-26T02:25:54.457784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422878376,"num_objects":51627,"num_object_clones":0,"num_object_copies":206508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51654,"num_write_kb":413114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51681,"ondisk_log_size":51681,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1296956,"kb_used_data":696028,"kb_used_omap":14,"kb_used_meta":600881,"kb_avail":376190404,"statfs":{"total":386547056640,"available":385218973696,"internally_reserved":0,"allocated":712732672,"data_stored":712028820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":615302707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,8,8,12,17,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35692544,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4357,"num_write_kb":34856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004215"},"pg_stats":[{"pgid":"1.f","version":"32'3083","reported_seq":3120,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.116678+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:50.116678+0000","last_peered":"2024-04-26T02:25:50.116678+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:50.116678+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25255936,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3338","reported_seq":3398,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.711185+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:53.711185+0000","last_peered":"2024-04-26T02:25:53.711185+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:53.711185+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3338,"log_dups_size":0,"ondisk_log_size":3338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308556,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3336,"num_write_kb":26684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3159","reported_seq":3232,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.720322+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:52.720322+0000","last_peered":"2024-04-26T02:25:52.720322+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:52.720322+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25852166,"num_objects":3156,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3122","reported_seq":3180,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.683760+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:53.683760+0000","last_peered":"2024-04-26T02:25:53.683760+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:53.683760+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539084,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":24949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3115","reported_seq":3166,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:49.966788+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:49.966788+0000","last_peered":"2024-04-26T02:25:49.966788+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:49.966788+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25489932,"num_objects":3112,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3219","reported_seq":3258,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.056656+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:50.056656+0000","last_peered":"2024-04-26T02:25:50.056656+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:50.056656+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3219,"log_dups_size":0,"ondisk_log_size":3219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26370048,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3295","reported_seq":5046,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.767220+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:53.767220+0000","last_peered":"2024-04-26T02:25:53.767220+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:53.767220+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939916,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3421","reported_seq":3482,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.802036+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:53.802036+0000","last_peered":"2024-04-26T02:25:53.802036+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:53.802036+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27980300,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":27340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3160","reported_seq":3212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:49.924892+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:49.924892+0000","last_peered":"2024-04-26T02:25:49.924892+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:49.924892+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3160,"log_dups_size":0,"ondisk_log_size":3160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25814040,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3238","reported_seq":3313,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.614225+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:52.614225+0000","last_peered":"2024-04-26T02:25:52.614225+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:52.614225+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3238,"log_dups_size":0,"ondisk_log_size":3238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26505740,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":25886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3146","reported_seq":3199,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.098191+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:50.098191+0000","last_peered":"2024-04-26T02:25:50.098191+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:50.098191+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3146,"log_dups_size":0,"ondisk_log_size":3146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25725714,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3143,"num_write_kb":25132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3161","reported_seq":3195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.096796+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:50.096796+0000","last_peered":"2024-04-26T02:25:50.096796+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:50.096796+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3161,"log_dups_size":0,"ondisk_log_size":3161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25884934,"num_objects":3160,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":25279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1172}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3314","reported_seq":3384,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.736570+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:52.736570+0000","last_peered":"2024-04-26T02:25:52.736570+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:52.736570+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138310,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3309","reported_seq":3357,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:50.032955+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:50.032955+0000","last_peered":"2024-04-26T02:25:50.032955+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:50.032955+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27070988,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1234}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3268","reported_seq":3342,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.686362+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:52.686362+0000","last_peered":"2024-04-26T02:25:52.686362+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:52.686362+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3268,"log_dups_size":0,"ondisk_log_size":3268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743308,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1212}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3333","reported_seq":3391,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.747444+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:53.747444+0000","last_peered":"2024-04-26T02:25:53.747444+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:53.747444+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3333,"log_dups_size":0,"ondisk_log_size":3333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27259404,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":26636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422878376,"num_objects":51627,"num_object_clones":0,"num_object_copies":206508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51654,"num_write_kb":413114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715870208,"data_stored":715870208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51681,"ondisk_log_size":51681,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374812,"kb_used_data":202900,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93997028,"statfs":{"total":96636764160,"available":96252956672,"internally_reserved":0,"allocated":207769600,"data_stored":207593637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,2,6,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378300,"kb_used_data":206900,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93993540,"statfs":{"total":96636764160,"available":96249384960,"internally_reserved":0,"allocated":211865600,"data_stored":211689637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159348,"kb_used_data":77740,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94212492,"statfs":{"total":96636764160,"available":96473591808,"internally_reserved":0,"allocated":79605760,"data_stored":79429797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384496,"kb_used_data":208488,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93987344,"statfs":{"total":96636764160,"available":96243040256,"internally_reserved":0,"allocated":213491712,"data_stored":213315749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213921792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80465920,"data_stored":80465920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212631552,"data_stored":212631552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208850944,"data_stored":208850944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:55.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:25:55.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3083", 'reported_seq': 3120, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:50.116678+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:25:50.116678+0000', 'last_peered': '2024-04-26T02:25:50.116678+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:25:50.116678+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3083, 'log_dups_size': 0, 'ondisk_log_size': 3083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25255936, 'num_objects': 3083, 'num_object_clones': 0, 'num_object_copies': 12332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3083, 'num_write_kb': 24664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3338", 'reported_seq': 3398, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:53.711185+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:53.711185+0000', 'last_peered': '2024-04-26T02:25:53.711185+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:53.711185+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3338, 'log_dups_size': 0, 'ondisk_log_size': 3338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27308556, 'num_objects': 3334, 'num_object_clones': 0, 'num_object_copies': 13336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3336, 'num_write_kb': 26684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3159", 'reported_seq': 3232, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:52.720322+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:52.720322+0000', 'last_peered': '2024-04-26T02:25:52.720322+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:52.720322+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3159, 'log_dups_size': 0, 'ondisk_log_size': 3159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25852166, 'num_objects': 3156, 'num_object_clones': 0, 'num_object_copies': 12624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3158, 'num_write_kb': 25254, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3122", 'reported_seq': 3180, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:53.683760+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:53.683760+0000', 'last_peered': '2024-04-26T02:25:53.683760+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:53.683760+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3122, 'log_dups_size': 0, 'ondisk_log_size': 3122, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25539084, 'num_objects': 3118, 'num_object_clones': 0, 'num_object_copies': 12472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3120, 'num_write_kb': 24949, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:55.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3115", 'reported_seq': 3166, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:49.966788+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:25:49.966788+0000', 'last_peered': '2024-04-26T02:25:49.966788+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:25:49.966788+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3115, 'log_dups_size': 0, 'ondisk_log_size': 3115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25489932, 'num_objects': 3112, 'num_object_clones': 0, 'num_object_copies': 12448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3113, 'num_write_kb': 24901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1101}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3219", 'reported_seq': 3258, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:50.056656+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:25:50.056656+0000', 'last_peered': '2024-04-26T02:25:50.056656+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:25:50.056656+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3219, 'log_dups_size': 0, 'ondisk_log_size': 3219, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26370048, 'num_objects': 3219, 'num_object_clones': 0, 'num_object_copies': 12876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3219, 'num_write_kb': 25752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3295", 'reported_seq': 5046, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:53.767220+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:53.767220+0000', 'last_peered': '2024-04-26T02:25:53.767220+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:53.767220+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3295, 'log_dups_size': 0, 'ondisk_log_size': 3295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26939916, 'num_objects': 3289, 'num_object_clones': 0, 'num_object_copies': 13156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3293, 'num_write_kb': 26324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 846, 'num_bytes_recovered': 6914048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2051}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3421", 'reported_seq': 3482, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:53.802036+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:53.802036+0000', 'last_peered': '2024-04-26T02:25:53.802036+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:53.802036+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3421, 'log_dups_size': 0, 'ondisk_log_size': 3421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27980300, 'num_objects': 3416, 'num_object_clones': 0, 'num_object_copies': 13664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3419, 'num_write_kb': 27340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3160", 'reported_seq': 3212, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:49.924892+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:25:49.924892+0000', 'last_peered': '2024-04-26T02:25:49.924892+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:25:49.924892+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3160, 'log_dups_size': 0, 'ondisk_log_size': 3160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25814040, 'num_objects': 3152, 'num_object_clones': 0, 'num_object_copies': 12608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3157, 'num_write_kb': 25233, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1148}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3238", 'reported_seq': 3313, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:52.614225+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:52.614225+0000', 'last_peered': '2024-04-26T02:25:52.614225+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:52.614225+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3238, 'log_dups_size': 0, 'ondisk_log_size': 3238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26505740, 'num_objects': 3236, 'num_object_clones': 0, 'num_object_copies': 12944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3236, 'num_write_kb': 25886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1199}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3146", 'reported_seq': 3199, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:50.098191+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:25:50.098191+0000', 'last_peered': '2024-04-26T02:25:50.098191+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:25:50.098191+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3146, 'log_dups_size': 0, 'ondisk_log_size': 3146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25725714, 'num_objects': 3141, 'num_object_clones': 0, 'num_object_copies': 12564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3143, 'num_write_kb': 25132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3161", 'reported_seq': 3195, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:50.096796+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:25:50.096796+0000', 'last_peered': '2024-04-26T02:25:50.096796+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:25:50.096796+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3161, 'log_dups_size': 0, 'ondisk_log_size': 3161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25884934, 'num_objects': 3160, 'num_object_clones': 0, 'num_object_copies': 12640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3160, 'num_write_kb': 25279, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1172}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3314", 'reported_seq': 3384, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:52.736570+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:52.736570+0000', 'last_peered': '2024-04-26T02:25:52.736570+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:52.736570+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3314, 'log_dups_size': 0, 'ondisk_log_size': 3314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27138310, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3313, 'num_write_kb': 26503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3309", 'reported_seq': 3357, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:50.032955+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:25:50.032955+0000', 'last_peered': '2024-04-26T02:25:50.032955+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:25:50.032955+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3309, 'log_dups_size': 0, 'ondisk_log_size': 3309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27070988, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3307, 'num_write_kb': 26452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1234}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:25:55.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:25:55.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3268", 'reported_seq': 3342, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:52.686362+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:52.686362+0000', 'last_peered': '2024-04-26T02:25:52.686362+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:52.686362+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3268, 'log_dups_size': 0, 'ondisk_log_size': 3268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26743308, 'num_objects': 3265, 'num_object_clones': 0, 'num_object_copies': 13060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3266, 'num_write_kb': 26125, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:25:55.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:25:55.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3333", 'reported_seq': 3391, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:53.747444+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:53.747444+0000', 'last_peered': '2024-04-26T02:25:53.747444+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:53.747444+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3333, 'log_dups_size': 0, 'ondisk_log_size': 3333, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27259404, 'num_objects': 3328, 'num_object_clones': 0, 'num_object_copies': 13312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3331, 'num_write_kb': 26636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:25:55.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:25:55.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 94 20 52832 52812 4.3876 2.42188 0.0905164 0.0355942 2024-04-26T02:25:56.544 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:56.544 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31896/206508 objects degraded (15.445%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":153,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51627,"data_bytes":422878376,"bytes_used":1328082944,"bytes_avail":385218973696,"bytes_total":386547056640,"degraded_objects":31896,"degraded_total":206508,"degraded_ratio":0.15445406473356965,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":173336,"recovering_keys_per_sec":0,"num_objects_recovered":254,"num_bytes_recovered":2080768,"num_keys_recovered":0,"write_bytes_sec":2973334,"read_op_per_sec":0,"write_op_per_sec":362},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T02:25:16.440230+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:25:56.622 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:25:56.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:56.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 95 20 53161 53141 4.36846 2.57031 0.0188406 0.0357475 2024-04-26T02:25:57.713 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 96 20 53499 53479 4.35046 2.64062 0.0599928 0.0359003 2024-04-26T02:25:57.922 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:25:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:25:57.939 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-04-26T02:25:56.458279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428891304,"num_objects":52361,"num_object_clones":0,"num_object_copies":209444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52388,"num_write_kb":418986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52415,"ondisk_log_size":52415,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1313372,"kb_used_data":704124,"kb_used_omap":14,"kb_used_meta":609201,"kb_avail":376173988,"statfs":{"total":386547056640,"available":385202163712,"internally_reserved":0,"allocated":721022976,"data_stored":720319124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":623822387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,8,10,8,17,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33996800,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004190"},"pg_stats":[{"pgid":"1.f","version":"32'3190","reported_seq":3227,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.162757+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:25:55.162757+0000","last_peered":"2024-04-26T02:25:55.162757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:25:55.162757+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26132480,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":25520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3338","reported_seq":3398,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.711185+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:53.711185+0000","last_peered":"2024-04-26T02:25:53.711185+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:53.711185+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3338,"log_dups_size":0,"ondisk_log_size":3338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308556,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3336,"num_write_kb":26684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3159","reported_seq":3232,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.720322+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:52.720322+0000","last_peered":"2024-04-26T02:25:52.720322+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:52.720322+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25852166,"num_objects":3156,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3122","reported_seq":3180,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.683760+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:53.683760+0000","last_peered":"2024-04-26T02:25:53.683760+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:53.683760+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539084,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":24949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3216","reported_seq":3267,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.000194+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:25:55.000194+0000","last_peered":"2024-04-26T02:25:55.000194+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:25:55.000194+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26317324,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":25709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3326","reported_seq":3365,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.100268+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:25:55.100268+0000","last_peered":"2024-04-26T02:25:55.100268+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:25:55.100268+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3295","reported_seq":5046,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.767220+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:53.767220+0000","last_peered":"2024-04-26T02:25:53.767220+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:53.767220+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26939916,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2051}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3421","reported_seq":3482,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.802036+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:53.802036+0000","last_peered":"2024-04-26T02:25:53.802036+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:53.802036+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27980300,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":27340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3265","reported_seq":3317,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:54.911555+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:25:54.911555+0000","last_peered":"2024-04-26T02:25:54.911555+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:25:54.911555+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26674200,"num_objects":3257,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3238","reported_seq":3313,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.614225+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:52.614225+0000","last_peered":"2024-04-26T02:25:52.614225+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:52.614225+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3238,"log_dups_size":0,"ondisk_log_size":3238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26505740,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":25886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3244","reported_seq":3297,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.193299+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:25:55.193299+0000","last_peered":"2024-04-26T02:25:55.193299+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:25:55.193299+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26528530,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3257","reported_seq":3291,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.112847+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:25:55.112847+0000","last_peered":"2024-04-26T02:25:55.112847+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:25:55.112847+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26671366,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":26047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1268}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3314","reported_seq":3384,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.736570+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:52.736570+0000","last_peered":"2024-04-26T02:25:52.736570+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:52.736570+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138310,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3429","reported_seq":3477,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:55.151326+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:25:55.151326+0000","last_peered":"2024-04-26T02:25:55.151326+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:25:55.151326+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054028,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1354}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3268","reported_seq":3342,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:52.686362+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:52.686362+0000","last_peered":"2024-04-26T02:25:52.686362+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:52.686362+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3268,"log_dups_size":0,"ondisk_log_size":3268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26743308,"num_objects":3265,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1212}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3333","reported_seq":3391,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:53.747444+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:53.747444+0000","last_peered":"2024-04-26T02:25:53.747444+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:53.747444+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3333,"log_dups_size":0,"ondisk_log_size":3333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27259404,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":26636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428891304,"num_objects":52361,"num_object_clones":0,"num_object_copies":209444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52388,"num_write_kb":418986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":6914048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":722849792,"data_stored":722849792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52415,"ondisk_log_size":52415,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391228,"kb_used_data":210996,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93980612,"statfs":{"total":96636764160,"available":96236146688,"internally_reserved":0,"allocated":216059904,"data_stored":215883941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378300,"kb_used_data":206900,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93993540,"statfs":{"total":96636764160,"available":96249384960,"internally_reserved":0,"allocated":211865600,"data_stored":211689637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159348,"kb_used_data":77740,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94212492,"statfs":{"total":96636764160,"available":96473591808,"internally_reserved":0,"allocated":79605760,"data_stored":79429797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384496,"kb_used_data":208488,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93987344,"statfs":{"total":96636764160,"available":96243040256,"internally_reserved":0,"allocated":213491712,"data_stored":213315749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213921792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80465920,"data_stored":80465920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212631552,"data_stored":212631552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215830528,"data_stored":215830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:58.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 97 20 53803 53783 4.3301 2.375 0.0559069 0.0360591 2024-04-26T02:25:59.714 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 98 20 54093 54073 4.30904 2.26562 0.0230998 0.036248 2024-04-26T02:26:00.715 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 99 20 54418 54398 4.29115 2.53906 0.0621315 0.0363993 2024-04-26T02:26:00.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:01.716 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:26:01.716774+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0365401 2024-04-26T02:26:01.716 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:26:01.716 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 100 20 54752 54732 4.2743 2.60938 0.0246654 0.0365401 2024-04-26T02:26:02.718 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 101 20 55086 55066 4.25772 2.60938 0.0746491 0.0366812 2024-04-26T02:26:02.774 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:02.775 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:02.901 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-04-26T02:26:02.460070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442145960,"num_objects":53979,"num_object_clones":0,"num_object_copies":215916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54006,"num_write_kb":431930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54033,"ondisk_log_size":54033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1411124,"kb_used_data":736660,"kb_used_omap":14,"kb_used_meta":674417,"kb_avail":376076236,"statfs":{"total":386547056640,"available":385102065664,"internally_reserved":0,"allocated":754339840,"data_stored":753635988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":690603571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,10,3,7,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27320320,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":26680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003279"},"pg_stats":[{"pgid":"1.f","version":"32'3285","reported_seq":3322,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.286576+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:00.286576+0000","last_peered":"2024-04-26T02:26:00.286576+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:00.286576+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26910720,"num_objects":3285,"num_object_clones":0,"num_object_copies":13140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3285,"num_write_kb":26280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3423","reported_seq":3484,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.627189+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:58.627189+0000","last_peered":"2024-04-26T02:25:58.627189+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:58.627189+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28004876,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3257","reported_seq":3331,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.750848+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:57.750848+0000","last_peered":"2024-04-26T02:25:57.750848+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:57.750848+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26654982,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3230","reported_seq":3288,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.765921+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:58.765921+0000","last_peered":"2024-04-26T02:25:58.765921+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:58.765921+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3230,"log_dups_size":0,"ondisk_log_size":3230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26423820,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3313","reported_seq":3364,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.208257+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:00.208257+0000","last_peered":"2024-04-26T02:26:00.208257+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:00.208257+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27111948,"num_objects":3310,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":26485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3424","reported_seq":3463,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.281726+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:00.281726+0000","last_peered":"2024-04-26T02:26:00.281726+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:00.281726+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3424,"log_dups_size":0,"ondisk_log_size":3424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3412","reported_seq":5338,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.833192+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:58.833192+0000","last_peered":"2024-04-26T02:25:58.833192+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:58.833192+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27898380,"num_objects":3406,"num_object_clones":0,"num_object_copies":13624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2256}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3498","reported_seq":3559,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.573571+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:58.573571+0000","last_peered":"2024-04-26T02:25:58.573571+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:58.573571+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3498,"log_dups_size":0,"ondisk_log_size":3498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611084,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3370","reported_seq":3422,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.203993+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:00.203993+0000","last_peered":"2024-04-26T02:26:00.203993+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:00.203993+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3370,"log_dups_size":0,"ondisk_log_size":3370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27534360,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3367,"num_write_kb":26913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3340","reported_seq":3415,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.741972+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:57.741972+0000","last_peered":"2024-04-26T02:25:57.741972+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:57.741972+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27341324,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":26702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3365","reported_seq":3418,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.254965+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:00.254965+0000","last_peered":"2024-04-26T02:26:00.254965+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:00.254965+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3365,"log_dups_size":0,"ondisk_log_size":3365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27519762,"num_objects":3360,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":26884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3357","reported_seq":3391,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.171964+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:00.171964+0000","last_peered":"2024-04-26T02:26:00.171964+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:00.171964+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27490566,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":26847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1368}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3411","reported_seq":3481,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.734514+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:57.734514+0000","last_peered":"2024-04-26T02:25:57.734514+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:57.734514+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27932934,"num_objects":3410,"num_object_clones":0,"num_object_copies":13640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3532","reported_seq":3580,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.245125+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:00.245125+0000","last_peered":"2024-04-26T02:26:00.245125+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:00.245125+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28897804,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3530,"num_write_kb":28236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3381","reported_seq":3455,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.712552+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:57.712552+0000","last_peered":"2024-04-26T02:25:57.712552+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:57.712552+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27669004,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3379,"num_write_kb":27029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3435","reported_seq":3495,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.812277+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:58.812277+0000","last_peered":"2024-04-26T02:25:58.812277+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:58.812277+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28094988,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":27452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442145960,"num_objects":53979,"num_object_clones":0,"num_object_copies":215916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54006,"num_write_kb":431930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":756551680,"data_stored":756551680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54033,"ondisk_log_size":54033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401436,"kb_used_data":217044,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93970404,"statfs":{"total":96636764160,"available":96225693696,"internally_reserved":0,"allocated":222253056,"data_stored":222077093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393000,"kb_used_data":213280,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93978840,"statfs":{"total":96636764160,"available":96234332160,"internally_reserved":0,"allocated":218398720,"data_stored":218222757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221748,"kb_used_data":91564,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94150092,"statfs":{"total":96636764160,"available":96409694208,"internally_reserved":0,"allocated":93761536,"data_stored":93585573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394940,"kb_used_data":214772,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93976900,"statfs":{"total":96636764160,"available":96232345600,"internally_reserved":0,"allocated":219926528,"data_stored":219750565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220512256,"data_stored":220512256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94494720,"data_stored":94494720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219197440,"data_stored":219197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222347264,"data_stored":222347264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:02.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:03.718 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 102 20 55379 55359 4.23842 2.28906 0.026244 0.0368514 2024-04-26T02:26:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:03.836 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:04.006 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-04-26T02:26:02.460070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442145960,"num_objects":53979,"num_object_clones":0,"num_object_copies":215916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54006,"num_write_kb":431930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54033,"ondisk_log_size":54033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1411124,"kb_used_data":736660,"kb_used_omap":14,"kb_used_meta":674417,"kb_avail":376076236,"statfs":{"total":386547056640,"available":385102065664,"internally_reserved":0,"allocated":754339840,"data_stored":753635988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":690603571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,10,3,7,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27320320,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":26680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003279"},"pg_stats":[{"pgid":"1.f","version":"32'3285","reported_seq":3322,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.286576+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:00.286576+0000","last_peered":"2024-04-26T02:26:00.286576+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:00.286576+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26910720,"num_objects":3285,"num_object_clones":0,"num_object_copies":13140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3285,"num_write_kb":26280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3423","reported_seq":3484,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.627189+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:25:58.627189+0000","last_peered":"2024-04-26T02:25:58.627189+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:25:58.627189+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28004876,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3257","reported_seq":3331,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.750848+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:25:57.750848+0000","last_peered":"2024-04-26T02:25:57.750848+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:25:57.750848+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26654982,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3230","reported_seq":3288,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.765921+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:25:58.765921+0000","last_peered":"2024-04-26T02:25:58.765921+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:25:58.765921+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3230,"log_dups_size":0,"ondisk_log_size":3230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26423820,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3313","reported_seq":3364,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.208257+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:00.208257+0000","last_peered":"2024-04-26T02:26:00.208257+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:00.208257+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27111948,"num_objects":3310,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":26485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3424","reported_seq":3463,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.281726+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:00.281726+0000","last_peered":"2024-04-26T02:26:00.281726+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:00.281726+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3424,"log_dups_size":0,"ondisk_log_size":3424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3412","reported_seq":5338,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.833192+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:25:58.833192+0000","last_peered":"2024-04-26T02:25:58.833192+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:25:58.833192+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27898380,"num_objects":3406,"num_object_clones":0,"num_object_copies":13624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2256}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3498","reported_seq":3559,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.573571+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:25:58.573571+0000","last_peered":"2024-04-26T02:25:58.573571+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:25:58.573571+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3498,"log_dups_size":0,"ondisk_log_size":3498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611084,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3370","reported_seq":3422,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.203993+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:00.203993+0000","last_peered":"2024-04-26T02:26:00.203993+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:00.203993+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3370,"log_dups_size":0,"ondisk_log_size":3370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27534360,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3367,"num_write_kb":26913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3340","reported_seq":3415,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.741972+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:25:57.741972+0000","last_peered":"2024-04-26T02:25:57.741972+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:25:57.741972+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27341324,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":26702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3365","reported_seq":3418,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.254965+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:00.254965+0000","last_peered":"2024-04-26T02:26:00.254965+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:00.254965+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3365,"log_dups_size":0,"ondisk_log_size":3365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27519762,"num_objects":3360,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":26884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3357","reported_seq":3391,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.171964+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:00.171964+0000","last_peered":"2024-04-26T02:26:00.171964+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:00.171964+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27490566,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":26847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1368}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3411","reported_seq":3481,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.734514+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:25:57.734514+0000","last_peered":"2024-04-26T02:25:57.734514+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:25:57.734514+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27932934,"num_objects":3410,"num_object_clones":0,"num_object_copies":13640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3532","reported_seq":3580,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:00.245125+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:00.245125+0000","last_peered":"2024-04-26T02:26:00.245125+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:00.245125+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28897804,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3530,"num_write_kb":28236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3381","reported_seq":3455,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:57.712552+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:25:57.712552+0000","last_peered":"2024-04-26T02:25:57.712552+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:25:57.712552+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27669004,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3379,"num_write_kb":27029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3435","reported_seq":3495,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:25:58.812277+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:25:58.812277+0000","last_peered":"2024-04-26T02:25:58.812277+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:25:58.812277+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28094988,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":27452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442145960,"num_objects":53979,"num_object_clones":0,"num_object_copies":215916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54006,"num_write_kb":431930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":7634944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":756551680,"data_stored":756551680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54033,"ondisk_log_size":54033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401436,"kb_used_data":217044,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93970404,"statfs":{"total":96636764160,"available":96225693696,"internally_reserved":0,"allocated":222253056,"data_stored":222077093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,2,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393000,"kb_used_data":213280,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93978840,"statfs":{"total":96636764160,"available":96234332160,"internally_reserved":0,"allocated":218398720,"data_stored":218222757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221748,"kb_used_data":91564,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94150092,"statfs":{"total":96636764160,"available":96409694208,"internally_reserved":0,"allocated":93761536,"data_stored":93585573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394940,"kb_used_data":214772,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93976900,"statfs":{"total":96636764160,"available":96232345600,"internally_reserved":0,"allocated":219926528,"data_stored":219750565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,1,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220512256,"data_stored":220512256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94494720,"data_stored":94494720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219197440,"data_stored":219197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222347264,"data_stored":222347264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3285", 'reported_seq': 3322, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.286576+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:00.286576+0000', 'last_peered': '2024-04-26T02:26:00.286576+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:00.286576+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3285, 'log_dups_size': 0, 'ondisk_log_size': 3285, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26910720, 'num_objects': 3285, 'num_object_clones': 0, 'num_object_copies': 13140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3285, 'num_write_kb': 26280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3423", 'reported_seq': 3484, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:58.627189+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:25:58.627189+0000', 'last_peered': '2024-04-26T02:25:58.627189+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:25:58.627189+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3423, 'log_dups_size': 0, 'ondisk_log_size': 3423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28004876, 'num_objects': 3419, 'num_object_clones': 0, 'num_object_copies': 13676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3421, 'num_write_kb': 27364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3257", 'reported_seq': 3331, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:57.750848+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:25:57.750848+0000', 'last_peered': '2024-04-26T02:25:57.750848+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:25:57.750848+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3257, 'log_dups_size': 0, 'ondisk_log_size': 3257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26654982, 'num_objects': 3254, 'num_object_clones': 0, 'num_object_copies': 13016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3256, 'num_write_kb': 26038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:04.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3230", 'reported_seq': 3288, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:58.765921+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:25:58.765921+0000', 'last_peered': '2024-04-26T02:25:58.765921+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:25:58.765921+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3230, 'log_dups_size': 0, 'ondisk_log_size': 3230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26423820, 'num_objects': 3226, 'num_object_clones': 0, 'num_object_copies': 12904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3228, 'num_write_kb': 25813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3313", 'reported_seq': 3364, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.208257+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:00.208257+0000', 'last_peered': '2024-04-26T02:26:00.208257+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:00.208257+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3313, 'log_dups_size': 0, 'ondisk_log_size': 3313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27111948, 'num_objects': 3310, 'num_object_clones': 0, 'num_object_copies': 13240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3311, 'num_write_kb': 26485, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3424", 'reported_seq': 3463, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.281726+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:00.281726+0000', 'last_peered': '2024-04-26T02:26:00.281726+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:00.281726+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3424, 'log_dups_size': 0, 'ondisk_log_size': 3424, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28049408, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3424, 'num_write_kb': 27392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1373}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3412", 'reported_seq': 5338, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:58.833192+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:25:58.833192+0000', 'last_peered': '2024-04-26T02:25:58.833192+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:25:58.833192+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3412, 'log_dups_size': 0, 'ondisk_log_size': 3412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27898380, 'num_objects': 3406, 'num_object_clones': 0, 'num_object_copies': 13624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1150, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3410, 'num_write_kb': 27260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 934, 'num_bytes_recovered': 7634944, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2256}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3498", 'reported_seq': 3559, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:58.573571+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:25:58.573571+0000', 'last_peered': '2024-04-26T02:25:58.573571+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:25:58.573571+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3498, 'log_dups_size': 0, 'ondisk_log_size': 3498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28611084, 'num_objects': 3493, 'num_object_clones': 0, 'num_object_copies': 13972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3496, 'num_write_kb': 27956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:04.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3370", 'reported_seq': 3422, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.203993+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:00.203993+0000', 'last_peered': '2024-04-26T02:26:00.203993+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:00.203993+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3370, 'log_dups_size': 0, 'ondisk_log_size': 3370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27534360, 'num_objects': 3362, 'num_object_clones': 0, 'num_object_copies': 13448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3367, 'num_write_kb': 26913, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1358}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3340", 'reported_seq': 3415, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:57.741972+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:25:57.741972+0000', 'last_peered': '2024-04-26T02:25:57.741972+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:25:57.741972+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3340, 'log_dups_size': 0, 'ondisk_log_size': 3340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27341324, 'num_objects': 3338, 'num_object_clones': 0, 'num_object_copies': 13352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3338, 'num_write_kb': 26702, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1301}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3365", 'reported_seq': 3418, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.254965+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:00.254965+0000', 'last_peered': '2024-04-26T02:26:00.254965+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:00.254965+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3365, 'log_dups_size': 0, 'ondisk_log_size': 3365, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27519762, 'num_objects': 3360, 'num_object_clones': 0, 'num_object_copies': 13440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3362, 'num_write_kb': 26884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1343}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3357", 'reported_seq': 3391, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.171964+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:00.171964+0000', 'last_peered': '2024-04-26T02:26:00.171964+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:00.171964+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3357, 'log_dups_size': 0, 'ondisk_log_size': 3357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27490566, 'num_objects': 3356, 'num_object_clones': 0, 'num_object_copies': 13424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3356, 'num_write_kb': 26847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1368}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:04.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3411", 'reported_seq': 3481, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:57.734514+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:25:57.734514+0000', 'last_peered': '2024-04-26T02:25:57.734514+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:25:57.734514+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3411, 'log_dups_size': 0, 'ondisk_log_size': 3411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27932934, 'num_objects': 3410, 'num_object_clones': 0, 'num_object_copies': 13640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3410, 'num_write_kb': 27279, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3532", 'reported_seq': 3580, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:00.245125+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:00.245125+0000', 'last_peered': '2024-04-26T02:26:00.245125+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:00.245125+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3532, 'log_dups_size': 0, 'ondisk_log_size': 3532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28897804, 'num_objects': 3528, 'num_object_clones': 0, 'num_object_copies': 14112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3530, 'num_write_kb': 28236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1457}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3381", 'reported_seq': 3455, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:57.712552+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:25:57.712552+0000', 'last_peered': '2024-04-26T02:25:57.712552+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:25:57.712552+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3381, 'log_dups_size': 0, 'ondisk_log_size': 3381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27669004, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 13512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3379, 'num_write_kb': 27029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:04.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3435", 'reported_seq': 3495, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:25:58.812277+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:25:58.812277+0000', 'last_peered': '2024-04-26T02:25:58.812277+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:25:58.812277+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3435, 'log_dups_size': 0, 'ondisk_log_size': 3435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28094988, 'num_objects': 3430, 'num_object_clones': 0, 'num_object_copies': 13720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3433, 'num_write_kb': 27452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:04.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:04.718 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 103 20 55694 55674 4.22117 2.46094 0.0231878 0.0369948 2024-04-26T02:26:05.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:05.569 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31692/219444 objects degraded (14.442%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":162,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54861,"data_bytes":449371304,"bytes_used":1466589184,"bytes_avail":385080467456,"bytes_total":386547056640,"degraded_objects":31692,"degraded_total":219444,"degraded_ratio":0.14441953300158583,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":202696,"recovering_keys_per_sec":0,"num_objects_recovered":299,"num_bytes_recovered":2433024,"num_keys_recovered":0,"write_bytes_sec":2878015,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-26T02:25:58.459103+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:05.654 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:05.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:05.719 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 104 20 56014 55994 4.20458 2.5 0.0372177 0.03714 2024-04-26T02:26:06.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 105 20 56344 56324 4.18902 2.57812 0.0298447 0.0372871 2024-04-26T02:26:06.976 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:06.977 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:06.992 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-04-26T02:26:06.461107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455105704,"num_objects":55561,"num_object_clones":0,"num_object_copies":222244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55588,"num_write_kb":444586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55615,"ondisk_log_size":55615,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1430272,"kb_used_data":755808,"kb_used_omap":14,"kb_used_meta":674417,"kb_avail":376057088,"statfs":{"total":386547056640,"available":385082458112,"internally_reserved":0,"allocated":773947392,"data_stored":773243540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":690603571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,17,12,9,8,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32227328,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":31472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"1.f","version":"32'3384","reported_seq":3421,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.357662+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:05.357662+0000","last_peered":"2024-04-26T02:26:05.357662+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:05.357662+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3512","reported_seq":3573,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:03.752522+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:03.752522+0000","last_peered":"2024-04-26T02:26:03.752522+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:03.752522+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28733964,"num_objects":3508,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3510,"num_write_kb":28076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3377","reported_seq":3451,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:02.690551+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:02.690551+0000","last_peered":"2024-04-26T02:26:02.690551+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:02.690551+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3377,"log_dups_size":0,"ondisk_log_size":3377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27638022,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":26998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3312","reported_seq":3368,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:03.808392+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:03.808392+0000","last_peered":"2024-04-26T02:26:03.808392+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:03.808392+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3312,"log_dups_size":0,"ondisk_log_size":3312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27095564,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":26469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3407","reported_seq":3458,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.222024+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:05.222024+0000","last_peered":"2024-04-26T02:26:05.222024+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:05.222024+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3407,"log_dups_size":0,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27881996,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3507","reported_seq":3546,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.093418+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:05.093418+0000","last_peered":"2024-04-26T02:26:05.093418+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:05.093418+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3516","reported_seq":5679,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:03.840861+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:03.840861+0000","last_peered":"2024-04-26T02:26:03.840861+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:03.840861+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28750348,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3514,"num_write_kb":28092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3609","reported_seq":3670,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:03.686554+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:03.686554+0000","last_peered":"2024-04-26T02:26:03.686554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:03.686554+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29520396,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":28844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3469","reported_seq":3521,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.265123+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:05.265123+0000","last_peered":"2024-04-26T02:26:05.265123+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:05.265123+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28345368,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3440","reported_seq":3515,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:02.629567+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:02.629567+0000","last_peered":"2024-04-26T02:26:02.629567+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:02.629567+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28160524,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3471","reported_seq":3524,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.259348+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:05.259348+0000","last_peered":"2024-04-26T02:26:05.259348+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:05.259348+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28388114,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3469","reported_seq":3503,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.349151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:05.349151+0000","last_peered":"2024-04-26T02:26:05.349151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:05.349151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28408070,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3499","reported_seq":3569,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:02.682202+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:02.682202+0000","last_peered":"2024-04-26T02:26:02.682202+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:02.682202+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28653830,"num_objects":3498,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":27983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1387}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3639","reported_seq":3687,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.302102+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:05.302102+0000","last_peered":"2024-04-26T02:26:05.302102+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:05.302102+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29774348,"num_objects":3635,"num_object_clones":0,"num_object_copies":14540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3476","reported_seq":3550,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:02.712090+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:02.712090+0000","last_peered":"2024-04-26T02:26:02.712090+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:02.712090+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3476,"log_dups_size":0,"ondisk_log_size":3476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28447244,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3474,"num_write_kb":27789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3528","reported_seq":3588,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:03.729440+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:03.729440+0000","last_peered":"2024-04-26T02:26:03.729440+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:03.729440+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3528,"log_dups_size":0,"ondisk_log_size":3528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28856844,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3526,"num_write_kb":28196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":455105704,"num_objects":55561,"num_object_clones":0,"num_object_copies":222244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55588,"num_write_kb":444586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":776171520,"data_stored":776171520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55615,"ondisk_log_size":55615,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399492,"kb_used_data":223420,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93972348,"statfs":{"total":96636764160,"available":96227684352,"internally_reserved":0,"allocated":228782080,"data_stored":228606117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,4,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403512,"kb_used_data":219632,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93968328,"statfs":{"total":96636764160,"available":96223567872,"internally_reserved":0,"allocated":224903168,"data_stored":224727205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221748,"kb_used_data":91564,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94150092,"statfs":{"total":96636764160,"available":96409694208,"internally_reserved":0,"allocated":93761536,"data_stored":93585573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405520,"kb_used_data":221192,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93966320,"statfs":{"total":96636764160,"available":96221511680,"internally_reserved":0,"allocated":226500608,"data_stored":226324645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188739279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,5,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227004416,"data_stored":227004416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94494720,"data_stored":94494720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225693696,"data_stored":225693696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":228978688,"data_stored":228978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:07.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 106 20 56590 56570 4.16764 1.92188 0.0114609 0.03747 2024-04-26T02:26:08.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 107 20 56893 56873 4.15082 2.36719 0.0894675 0.0376268 2024-04-26T02:26:09.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 108 20 57176 57156 4.13286 2.21094 0.0490225 0.0377878 2024-04-26T02:26:09.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:10.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 109 20 57497 57477 4.11795 2.50781 0.0465676 0.0379241 2024-04-26T02:26:11.333 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:11.333 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:11.424 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-26T02:26:10.462171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461970600,"num_objects":56399,"num_object_clones":0,"num_object_copies":225596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56426,"num_write_kb":451290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56453,"ondisk_log_size":56453,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1420284,"kb_used_data":774940,"kb_used_omap":14,"kb_used_meta":645297,"kb_avail":376067076,"statfs":{"total":386547056640,"available":385092685824,"internally_reserved":0,"allocated":793538560,"data_stored":792834708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":660784691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,9,9,9,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29720576,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":2637824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003153"},"pg_stats":[{"pgid":"1.f","version":"32'3384","reported_seq":3421,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.357662+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:05.357662+0000","last_peered":"2024-04-26T02:26:05.357662+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:05.357662+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3611","reported_seq":3672,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.752886+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:08.752886+0000","last_peered":"2024-04-26T02:26:08.752886+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:08.752886+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29544972,"num_objects":3607,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":28868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3485","reported_seq":3559,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.655267+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:07.655267+0000","last_peered":"2024-04-26T02:26:07.655267+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:07.655267+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28522758,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3418","reported_seq":3476,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.683633+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:08.683633+0000","last_peered":"2024-04-26T02:26:08.683633+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:08.683633+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27963916,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3407","reported_seq":3458,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.222024+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:05.222024+0000","last_peered":"2024-04-26T02:26:05.222024+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:05.222024+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3407,"log_dups_size":0,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27881996,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3507","reported_seq":3546,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.093418+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:05.093418+0000","last_peered":"2024-04-26T02:26:05.093418+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:05.093418+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3596","reported_seq":5995,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.853560+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:08.853560+0000","last_peered":"2024-04-26T02:26:08.853560+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:08.853560+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29405708,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3594,"num_write_kb":28732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1170,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3701","reported_seq":3762,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.688994+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:08.688994+0000","last_peered":"2024-04-26T02:26:08.688994+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:08.688994+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274060,"num_objects":3696,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1555}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3469","reported_seq":3521,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.265123+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:05.265123+0000","last_peered":"2024-04-26T02:26:05.265123+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:05.265123+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28345368,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3547","reported_seq":3622,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.766046+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:07.766046+0000","last_peered":"2024-04-26T02:26:07.766046+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:07.766046+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29037068,"num_objects":3545,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3471","reported_seq":3524,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.259348+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:05.259348+0000","last_peered":"2024-04-26T02:26:05.259348+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:05.259348+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28388114,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3469","reported_seq":3503,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.349151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:05.349151+0000","last_peered":"2024-04-26T02:26:05.349151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:05.349151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28408070,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3576","reported_seq":3646,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.721201+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:07.721201+0000","last_peered":"2024-04-26T02:26:07.721201+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:07.721201+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3576,"log_dups_size":0,"ondisk_log_size":3576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29284614,"num_objects":3575,"num_object_clones":0,"num_object_copies":14300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3639","reported_seq":3687,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.302102+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:05.302102+0000","last_peered":"2024-04-26T02:26:05.302102+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:05.302102+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29774348,"num_objects":3635,"num_object_clones":0,"num_object_copies":14540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3559","reported_seq":3633,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.599222+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:07.599222+0000","last_peered":"2024-04-26T02:26:07.599222+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:07.599222+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3559,"log_dups_size":0,"ondisk_log_size":3559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127180,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3614","reported_seq":3674,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.629624+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:08.629624+0000","last_peered":"2024-04-26T02:26:08.629624+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:08.629624+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29561356,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":28884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":461970600,"num_objects":56399,"num_object_clones":0,"num_object_copies":225596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56426,"num_write_kb":451290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":795389952,"data_stored":795389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56453,"ondisk_log_size":56453,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399492,"kb_used_data":223420,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93972348,"statfs":{"total":96636764160,"available":96227684352,"internally_reserved":0,"allocated":228782080,"data_stored":228606117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,4,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397236,"kb_used_data":225836,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93974604,"statfs":{"total":96636764160,"available":96229994496,"internally_reserved":0,"allocated":231256064,"data_stored":231080101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220244,"kb_used_data":98380,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94151596,"statfs":{"total":96636764160,"available":96411234304,"internally_reserved":0,"allocated":100741120,"data_stored":100565157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403312,"kb_used_data":227304,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93968528,"statfs":{"total":96636764160,"available":96223772672,"internally_reserved":0,"allocated":232759296,"data_stored":232583333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233074688,"data_stored":233074688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101494784,"data_stored":101494784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231841792,"data_stored":231841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":228978688,"data_stored":228978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:11.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:11.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 110 20 57802 57782 4.10218 2.38281 0.0366462 0.0380616 2024-04-26T02:26:11.936 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:11.936 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:12.020 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-26T02:26:10.462171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461970600,"num_objects":56399,"num_object_clones":0,"num_object_copies":225596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56426,"num_write_kb":451290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56453,"ondisk_log_size":56453,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1420284,"kb_used_data":774940,"kb_used_omap":14,"kb_used_meta":645297,"kb_avail":376067076,"statfs":{"total":386547056640,"available":385092685824,"internally_reserved":0,"allocated":793538560,"data_stored":792834708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":660784691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,9,9,9,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29720576,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":29024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":2637824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003153"},"pg_stats":[{"pgid":"1.f","version":"32'3384","reported_seq":3421,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.357662+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:05.357662+0000","last_peered":"2024-04-26T02:26:05.357662+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:05.357662+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3611","reported_seq":3672,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.752886+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:08.752886+0000","last_peered":"2024-04-26T02:26:08.752886+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:08.752886+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29544972,"num_objects":3607,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":28868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3485","reported_seq":3559,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.655267+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:07.655267+0000","last_peered":"2024-04-26T02:26:07.655267+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:07.655267+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28522758,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3418","reported_seq":3476,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.683633+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:08.683633+0000","last_peered":"2024-04-26T02:26:08.683633+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:08.683633+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27963916,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3407","reported_seq":3458,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.222024+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:05.222024+0000","last_peered":"2024-04-26T02:26:05.222024+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:05.222024+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3407,"log_dups_size":0,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27881996,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3507","reported_seq":3546,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.093418+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:05.093418+0000","last_peered":"2024-04-26T02:26:05.093418+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:05.093418+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3596","reported_seq":5995,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.853560+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:08.853560+0000","last_peered":"2024-04-26T02:26:08.853560+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:08.853560+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29405708,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3594,"num_write_kb":28732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1170,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3701","reported_seq":3762,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.688994+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:08.688994+0000","last_peered":"2024-04-26T02:26:08.688994+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:08.688994+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274060,"num_objects":3696,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1555}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3469","reported_seq":3521,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.265123+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:05.265123+0000","last_peered":"2024-04-26T02:26:05.265123+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:05.265123+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28345368,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3547","reported_seq":3622,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.766046+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:07.766046+0000","last_peered":"2024-04-26T02:26:07.766046+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:07.766046+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29037068,"num_objects":3545,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3471","reported_seq":3524,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.259348+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:05.259348+0000","last_peered":"2024-04-26T02:26:05.259348+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:05.259348+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28388114,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3469","reported_seq":3503,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.349151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:05.349151+0000","last_peered":"2024-04-26T02:26:05.349151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:05.349151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28408070,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3576","reported_seq":3646,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.721201+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:07.721201+0000","last_peered":"2024-04-26T02:26:07.721201+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:07.721201+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3576,"log_dups_size":0,"ondisk_log_size":3576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29284614,"num_objects":3575,"num_object_clones":0,"num_object_copies":14300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3639","reported_seq":3687,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:05.302102+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:05.302102+0000","last_peered":"2024-04-26T02:26:05.302102+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:05.302102+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29774348,"num_objects":3635,"num_object_clones":0,"num_object_copies":14540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3559","reported_seq":3633,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:07.599222+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:07.599222+0000","last_peered":"2024-04-26T02:26:07.599222+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:07.599222+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3559,"log_dups_size":0,"ondisk_log_size":3559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29127180,"num_objects":3556,"num_object_clones":0,"num_object_copies":14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3614","reported_seq":3674,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:08.629624+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:08.629624+0000","last_peered":"2024-04-26T02:26:08.629624+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:08.629624+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29561356,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":28884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":461970600,"num_objects":56399,"num_object_clones":0,"num_object_copies":225596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56426,"num_write_kb":451290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":795389952,"data_stored":795389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56453,"ondisk_log_size":56453,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399492,"kb_used_data":223420,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93972348,"statfs":{"total":96636764160,"available":96227684352,"internally_reserved":0,"allocated":228782080,"data_stored":228606117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,4,4,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397236,"kb_used_data":225836,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93974604,"statfs":{"total":96636764160,"available":96229994496,"internally_reserved":0,"allocated":231256064,"data_stored":231080101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220244,"kb_used_data":98380,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94151596,"statfs":{"total":96636764160,"available":96411234304,"internally_reserved":0,"allocated":100741120,"data_stored":100565157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403312,"kb_used_data":227304,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93968528,"statfs":{"total":96636764160,"available":96223772672,"internally_reserved":0,"allocated":232759296,"data_stored":232583333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180219599},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233074688,"data_stored":233074688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101494784,"data_stored":101494784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231841792,"data_stored":231841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":228978688,"data_stored":228978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3384", 'reported_seq': 3421, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.357662+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:05.357662+0000', 'last_peered': '2024-04-26T02:26:05.357662+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:05.357662+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3384, 'log_dups_size': 0, 'ondisk_log_size': 3384, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27721728, 'num_objects': 3384, 'num_object_clones': 0, 'num_object_copies': 13536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3384, 'num_write_kb': 27072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3611", 'reported_seq': 3672, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:08.752886+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:08.752886+0000', 'last_peered': '2024-04-26T02:26:08.752886+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:08.752886+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3611, 'log_dups_size': 0, 'ondisk_log_size': 3611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29544972, 'num_objects': 3607, 'num_object_clones': 0, 'num_object_copies': 14428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3609, 'num_write_kb': 28868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3485", 'reported_seq': 3559, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:07.655267+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:07.655267+0000', 'last_peered': '2024-04-26T02:26:07.655267+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:07.655267+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28522758, 'num_objects': 3482, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3484, 'num_write_kb': 27862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1458}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:12.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3418", 'reported_seq': 3476, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:08.683633+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:08.683633+0000', 'last_peered': '2024-04-26T02:26:08.683633+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:08.683633+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3418, 'log_dups_size': 0, 'ondisk_log_size': 3418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27963916, 'num_objects': 3414, 'num_object_clones': 0, 'num_object_copies': 13656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3416, 'num_write_kb': 27317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3407", 'reported_seq': 3458, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.222024+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:05.222024+0000', 'last_peered': '2024-04-26T02:26:05.222024+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:05.222024+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3407, 'log_dups_size': 0, 'ondisk_log_size': 3407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27881996, 'num_objects': 3404, 'num_object_clones': 0, 'num_object_copies': 13616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3405, 'num_write_kb': 27237, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3507", 'reported_seq': 3546, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.093418+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:05.093418+0000', 'last_peered': '2024-04-26T02:26:05.093418+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:05.093418+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3507, 'log_dups_size': 0, 'ondisk_log_size': 3507, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28729344, 'num_objects': 3507, 'num_object_clones': 0, 'num_object_copies': 14028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3507, 'num_write_kb': 28056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1456}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3596", 'reported_seq': 5995, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:08.853560+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:26:08.853560+0000', 'last_peered': '2024-04-26T02:26:08.853560+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:26:08.853560+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3596, 'log_dups_size': 0, 'ondisk_log_size': 3596, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29405708, 'num_objects': 3590, 'num_object_clones': 0, 'num_object_copies': 14360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 916, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3594, 'num_write_kb': 28732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1170, 'num_bytes_recovered': 9551872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2674}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3701", 'reported_seq': 3762, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:08.688994+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:08.688994+0000', 'last_peered': '2024-04-26T02:26:08.688994+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:08.688994+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3701, 'log_dups_size': 0, 'ondisk_log_size': 3701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30274060, 'num_objects': 3696, 'num_object_clones': 0, 'num_object_copies': 14784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3699, 'num_write_kb': 29580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1555}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3469", 'reported_seq': 3521, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.265123+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:05.265123+0000', 'last_peered': '2024-04-26T02:26:05.265123+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:05.265123+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3469, 'log_dups_size': 0, 'ondisk_log_size': 3469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28345368, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3466, 'num_write_kb': 27705, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1457}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.022 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3547", 'reported_seq': 3622, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:07.766046+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:07.766046+0000', 'last_peered': '2024-04-26T02:26:07.766046+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:07.766046+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3547, 'log_dups_size': 0, 'ondisk_log_size': 3547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29037068, 'num_objects': 3545, 'num_object_clones': 0, 'num_object_copies': 14180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3545, 'num_write_kb': 28358, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1508}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3471", 'reported_seq': 3524, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.259348+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:05.259348+0000', 'last_peered': '2024-04-26T02:26:05.259348+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:05.259348+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3471, 'log_dups_size': 0, 'ondisk_log_size': 3471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28388114, 'num_objects': 3466, 'num_object_clones': 0, 'num_object_copies': 13864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3468, 'num_write_kb': 27732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1449}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3469", 'reported_seq': 3503, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.349151+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:05.349151+0000', 'last_peered': '2024-04-26T02:26:05.349151+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:05.349151+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3469, 'log_dups_size': 0, 'ondisk_log_size': 3469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28408070, 'num_objects': 3468, 'num_object_clones': 0, 'num_object_copies': 13872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3468, 'num_write_kb': 27743, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1480}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3576", 'reported_seq': 3646, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:07.721201+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:07.721201+0000', 'last_peered': '2024-04-26T02:26:07.721201+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:07.721201+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3576, 'log_dups_size': 0, 'ondisk_log_size': 3576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29284614, 'num_objects': 3575, 'num_object_clones': 0, 'num_object_copies': 14300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3575, 'num_write_kb': 28599, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3639", 'reported_seq': 3687, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:05.302102+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:05.302102+0000', 'last_peered': '2024-04-26T02:26:05.302102+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:05.302102+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3639, 'log_dups_size': 0, 'ondisk_log_size': 3639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29774348, 'num_objects': 3635, 'num_object_clones': 0, 'num_object_copies': 14540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3637, 'num_write_kb': 29092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:12.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3559", 'reported_seq': 3633, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:07.599222+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:26:07.599222+0000', 'last_peered': '2024-04-26T02:26:07.599222+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:07.599222+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3559, 'log_dups_size': 0, 'ondisk_log_size': 3559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29127180, 'num_objects': 3556, 'num_object_clones': 0, 'num_object_copies': 14224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3557, 'num_write_kb': 28453, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:12.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:12.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3614", 'reported_seq': 3674, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:08.629624+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:08.629624+0000', 'last_peered': '2024-04-26T02:26:08.629624+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:08.629624+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3614, 'log_dups_size': 0, 'ondisk_log_size': 3614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29561356, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3612, 'num_write_kb': 28884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:12.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:12.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 111 20 58071 58051 4.08416 2.10156 0.0256014 0.0382387 2024-04-26T02:26:13.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:13.672 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31574/228208 objects degraded (13.836%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":170,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57052,"data_bytes":467319976,"bytes_used":1475543040,"bytes_avail":385071513600,"bytes_total":386547056640,"degraded_objects":31574,"degraded_total":228208,"degraded_ratio":0.13835623641590128,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":159701,"recovering_keys_per_sec":0,"num_objects_recovered":236,"num_bytes_recovered":1916928,"num_keys_recovered":0,"write_bytes_sec":2097275,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-26T02:25:58.459103+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:13.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 112 20 58382 58362 4.06939 2.42969 0.0200851 0.0383756 2024-04-26T02:26:13.799 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:13.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:14.721 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 113 20 58689 58669 4.05461 2.39844 0.0493293 0.0385175 2024-04-26T02:26:15.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:15.176 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:15.355 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-04-26T02:26:14.463227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":474324136,"num_objects":57907,"num_object_clones":0,"num_object_copies":231628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57934,"num_write_kb":463354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57961,"ondisk_log_size":57961,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1497160,"kb_used_data":799080,"kb_used_omap":14,"kb_used_meta":698033,"kb_avail":375990200,"statfs":{"total":386547056640,"available":385013964800,"internally_reserved":0,"allocated":818257920,"data_stored":817554068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":714786355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,10,5,13,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32178176,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3928,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":2850816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.f","version":"32'3490","reported_seq":3527,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.304268+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:10.304268+0000","last_peered":"2024-04-26T02:26:10.304268+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:10.304268+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3490,"log_dups_size":0,"ondisk_log_size":3490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28590080,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":27920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3695","reported_seq":3756,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.802908+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:13.802908+0000","last_peered":"2024-04-26T02:26:13.802908+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:13.802908+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233100,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":29540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3565","reported_seq":3639,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:12.728057+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:12.728057+0000","last_peered":"2024-04-26T02:26:12.728057+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:12.728057+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3565,"log_dups_size":0,"ondisk_log_size":3565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29178118,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3564,"num_write_kb":28502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1538}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3510","reported_seq":3569,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.859169+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:13.859169+0000","last_peered":"2024-04-26T02:26:13.859169+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:13.859169+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28717580,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3503","reported_seq":3554,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.280084+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:10.280084+0000","last_peered":"2024-04-26T02:26:10.280084+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:10.280084+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28668428,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3605","reported_seq":3644,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.414578+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:10.414578+0000","last_peered":"2024-04-26T02:26:10.414578+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:10.414578+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29532160,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":28840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3683","reported_seq":6311,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.850945+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:13.850945+0000","last_peered":"2024-04-26T02:26:13.850945+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:13.850945+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30118412,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3807","reported_seq":3868,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.710594+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:13.710594+0000","last_peered":"2024-04-26T02:26:13.710594+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:13.710594+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31142412,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":30428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3557","reported_seq":3609,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.412208+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:10.412208+0000","last_peered":"2024-04-26T02:26:10.412208+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:10.412208+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29066264,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3554,"num_write_kb":28409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3639","reported_seq":3713,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:12.771651+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:12.771651+0000","last_peered":"2024-04-26T02:26:12.771651+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:12.771651+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29790732,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3555","reported_seq":3608,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.294870+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:10.294870+0000","last_peered":"2024-04-26T02:26:10.294870+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:10.294870+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29076242,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3552,"num_write_kb":28404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3558","reported_seq":3592,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.423008+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:10.423008+0000","last_peered":"2024-04-26T02:26:10.423008+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:10.423008+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29137158,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":28455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3675","reported_seq":3746,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:12.793837+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:12.793837+0000","last_peered":"2024-04-26T02:26:12.793837+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:12.793837+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30095622,"num_objects":3674,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3674,"num_write_kb":29391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1563}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3731","reported_seq":3779,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:10.403289+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:10.403289+0000","last_peered":"2024-04-26T02:26:10.403289+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:10.403289+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30528012,"num_objects":3727,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":29828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1656}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3669","reported_seq":3743,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:12.733434+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:12.733434+0000","last_peered":"2024-04-26T02:26:12.733434+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:12.733434+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30028300,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":29333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1613}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3719","reported_seq":3779,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.817362+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:13.817362+0000","last_peered":"2024-04-26T02:26:13.817362+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:13.817362+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30421516,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":474324136,"num_objects":57907,"num_object_clones":0,"num_object_copies":231628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57934,"num_write_kb":463354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":820330496,"data_stored":820330496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57961,"ondisk_log_size":57961,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409572,"kb_used_data":229340,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93962268,"statfs":{"total":96636764160,"available":96217362432,"internally_reserved":0,"allocated":234844160,"data_stored":234668197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,0,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425284,"kb_used_data":231676,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93946556,"statfs":{"total":96636764160,"available":96201273344,"internally_reserved":0,"allocated":237236224,"data_stored":237060261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":230840,"kb_used_data":104816,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94141000,"statfs":{"total":96636764160,"available":96400384000,"internally_reserved":0,"allocated":107331584,"data_stored":107155621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431464,"kb_used_data":233248,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93940376,"statfs":{"total":96636764160,"available":96194945024,"internally_reserved":0,"allocated":238845952,"data_stored":238669989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239288320,"data_stored":239288320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107937792,"data_stored":107937792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237936640,"data_stored":237936640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":235167744,"data_stored":235167744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:15.722 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 114 20 58999 58979 4.04029 2.42188 0.109356 0.0386564 2024-04-26T02:26:16.722 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 115 20 59328 59308 4.02751 2.57031 0.0417286 0.0387791 2024-04-26T02:26:17.722 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 116 20 59663 59643 4.01535 2.61719 0.0734815 0.0388987 2024-04-26T02:26:18.357 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:18.722 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 117 20 59931 59911 3.99893 2.09375 0.0385837 0.0390551 2024-04-26T02:26:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:19.435 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:19.452 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-26T02:26:18.465350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483237032,"num_objects":58995,"num_object_clones":0,"num_object_copies":235980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59022,"num_write_kb":472058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59049,"ondisk_log_size":59049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1568644,"kb_used_data":817828,"kb_used_omap":14,"kb_used_meta":750769,"kb_avail":375918716,"statfs":{"total":386547056640,"available":384940765184,"internally_reserved":0,"allocated":837455872,"data_stored":836752020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":768788019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,13,11,11,12,9,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28131328,"num_objects":3434,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":27472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"1.f","version":"32'3587","reported_seq":3624,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.460042+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:15.460042+0000","last_peered":"2024-04-26T02:26:15.460042+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:15.460042+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29384704,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3695","reported_seq":3756,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.802908+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:13.802908+0000","last_peered":"2024-04-26T02:26:13.802908+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:13.802908+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233100,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":29540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3666","reported_seq":3740,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.727723+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:17.727723+0000","last_peered":"2024-04-26T02:26:17.727723+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:17.727723+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30005510,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3510","reported_seq":3569,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.859169+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:13.859169+0000","last_peered":"2024-04-26T02:26:13.859169+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:13.859169+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28717580,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3580","reported_seq":3632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.454438+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:15.454438+0000","last_peered":"2024-04-26T02:26:15.454438+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:15.454438+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29299212,"num_objects":3577,"num_object_clones":0,"num_object_copies":14308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3694","reported_seq":3733,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.328109+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:15.328109+0000","last_peered":"2024-04-26T02:26:15.328109+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:15.328109+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":29552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3683","reported_seq":6311,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.850945+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:13.850945+0000","last_peered":"2024-04-26T02:26:13.850945+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:13.850945+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30118412,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3807","reported_seq":3868,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.710594+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:13.710594+0000","last_peered":"2024-04-26T02:26:13.710594+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:13.710594+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31142412,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":30428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3653","reported_seq":3705,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.512051+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:15.512051+0000","last_peered":"2024-04-26T02:26:15.512051+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:15.512051+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29852696,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1641}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3753","reported_seq":3828,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.783035+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:17.783035+0000","last_peered":"2024-04-26T02:26:17.783035+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:17.783035+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30724620,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3650","reported_seq":3703,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.188157+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:15.188157+0000","last_peered":"2024-04-26T02:26:15.188157+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:15.188157+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29854482,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":29164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3667","reported_seq":3700,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.488216+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:15.488216+0000","last_peered":"2024-04-26T02:26:15.488216+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:15.488216+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30030086,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3666,"num_write_kb":29327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1678}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3764","reported_seq":3835,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.619200+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:17.619200+0000","last_peered":"2024-04-26T02:26:17.619200+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:17.619200+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30824710,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3837","reported_seq":3884,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.359116+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:15.359116+0000","last_peered":"2024-04-26T02:26:15.359116+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:15.359116+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31396364,"num_objects":3833,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":30676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3784","reported_seq":3856,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.757268+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:17.757268+0000","last_peered":"2024-04-26T02:26:17.757268+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:17.757268+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30970380,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3719","reported_seq":3779,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.817362+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:13.817362+0000","last_peered":"2024-04-26T02:26:13.817362+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:13.817362+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30421516,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483237032,"num_objects":58995,"num_object_clones":0,"num_object_copies":235980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59022,"num_write_kb":472058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":839884800,"data_stored":839884800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59049,"ondisk_log_size":59049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437848,"kb_used_data":235408,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93933992,"statfs":{"total":96636764160,"available":96188407808,"internally_reserved":0,"allocated":241057792,"data_stored":240881829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":207285968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,2,4,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457868,"kb_used_data":237892,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93913972,"statfs":{"total":96636764160,"available":96167907328,"internally_reserved":0,"allocated":243601408,"data_stored":243425445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241464,"kb_used_data":111280,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94130376,"statfs":{"total":96636764160,"available":96389505024,"internally_reserved":0,"allocated":113950720,"data_stored":113774757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431464,"kb_used_data":233248,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93940376,"statfs":{"total":96636764160,"available":96194945024,"internally_reserved":0,"allocated":238845952,"data_stored":238669989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239288320,"data_stored":239288320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114724864,"data_stored":114724864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":244461568,"data_stored":244461568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241410048,"data_stored":241410048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:19.453 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:19.723 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 118 20 60203 60183 3.98303 2.125 0.137989 0.0392098 2024-04-26T02:26:20.187 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:20.187 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:20.203 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-26T02:26:18.465350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":483237032,"num_objects":58995,"num_object_clones":0,"num_object_copies":235980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59022,"num_write_kb":472058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59049,"ondisk_log_size":59049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1568644,"kb_used_data":817828,"kb_used_omap":14,"kb_used_meta":750769,"kb_avail":375918716,"statfs":{"total":386547056640,"available":384940765184,"internally_reserved":0,"allocated":837455872,"data_stored":836752020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":768788019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,13,11,11,12,9,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28131328,"num_objects":3434,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":27472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"1.f","version":"32'3587","reported_seq":3624,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.460042+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:15.460042+0000","last_peered":"2024-04-26T02:26:15.460042+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:15.460042+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29384704,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3695","reported_seq":3756,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.802908+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:13.802908+0000","last_peered":"2024-04-26T02:26:13.802908+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:13.802908+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30233100,"num_objects":3691,"num_object_clones":0,"num_object_copies":14764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":29540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3666","reported_seq":3740,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.727723+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:17.727723+0000","last_peered":"2024-04-26T02:26:17.727723+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:17.727723+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30005510,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3510","reported_seq":3569,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.859169+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:13.859169+0000","last_peered":"2024-04-26T02:26:13.859169+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:13.859169+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28717580,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3508,"num_write_kb":28053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3580","reported_seq":3632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.454438+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:15.454438+0000","last_peered":"2024-04-26T02:26:15.454438+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:15.454438+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29299212,"num_objects":3577,"num_object_clones":0,"num_object_copies":14308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3694","reported_seq":3733,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.328109+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:15.328109+0000","last_peered":"2024-04-26T02:26:15.328109+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:15.328109+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":29552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3683","reported_seq":6311,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.850945+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:13.850945+0000","last_peered":"2024-04-26T02:26:13.850945+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:13.850945+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30118412,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3807","reported_seq":3868,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.710594+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:13.710594+0000","last_peered":"2024-04-26T02:26:13.710594+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:13.710594+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31142412,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3805,"num_write_kb":30428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3653","reported_seq":3705,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.512051+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:15.512051+0000","last_peered":"2024-04-26T02:26:15.512051+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:15.512051+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29852696,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1641}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3753","reported_seq":3828,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.783035+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:17.783035+0000","last_peered":"2024-04-26T02:26:17.783035+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:17.783035+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30724620,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3650","reported_seq":3703,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.188157+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:15.188157+0000","last_peered":"2024-04-26T02:26:15.188157+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:15.188157+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29854482,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":29164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3667","reported_seq":3700,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.488216+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:15.488216+0000","last_peered":"2024-04-26T02:26:15.488216+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:15.488216+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30030086,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3666,"num_write_kb":29327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1678}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3764","reported_seq":3835,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.619200+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:17.619200+0000","last_peered":"2024-04-26T02:26:17.619200+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:17.619200+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30824710,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3837","reported_seq":3884,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:15.359116+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:15.359116+0000","last_peered":"2024-04-26T02:26:15.359116+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:15.359116+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31396364,"num_objects":3833,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":30676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3784","reported_seq":3856,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.757268+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:17.757268+0000","last_peered":"2024-04-26T02:26:17.757268+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:17.757268+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30970380,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3719","reported_seq":3779,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:13.817362+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:13.817362+0000","last_peered":"2024-04-26T02:26:13.817362+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:13.817362+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30421516,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":483237032,"num_objects":58995,"num_object_clones":0,"num_object_copies":235980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59022,"num_write_kb":472058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":839884800,"data_stored":839884800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59049,"ondisk_log_size":59049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437848,"kb_used_data":235408,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93933992,"statfs":{"total":96636764160,"available":96188407808,"internally_reserved":0,"allocated":241057792,"data_stored":240881829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":207285968},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,2,4,1,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457868,"kb_used_data":237892,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93913972,"statfs":{"total":96636764160,"available":96167907328,"internally_reserved":0,"allocated":243601408,"data_stored":243425445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241464,"kb_used_data":111280,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94130376,"statfs":{"total":96636764160,"available":96389505024,"internally_reserved":0,"allocated":113950720,"data_stored":113774757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431464,"kb_used_data":233248,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93940376,"statfs":{"total":96636764160,"available":96194945024,"internally_reserved":0,"allocated":238845952,"data_stored":238669989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239288320,"data_stored":239288320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114724864,"data_stored":114724864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":244461568,"data_stored":244461568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241410048,"data_stored":241410048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:20.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:20.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3587", 'reported_seq': 3624, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.460042+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:15.460042+0000', 'last_peered': '2024-04-26T02:26:15.460042+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:15.460042+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3587, 'log_dups_size': 0, 'ondisk_log_size': 3587, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29384704, 'num_objects': 3587, 'num_object_clones': 0, 'num_object_copies': 14348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3587, 'num_write_kb': 28696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:20.204 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3695", 'reported_seq': 3756, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:13.802908+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:13.802908+0000', 'last_peered': '2024-04-26T02:26:13.802908+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:13.802908+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3695, 'log_dups_size': 0, 'ondisk_log_size': 3695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30233100, 'num_objects': 3691, 'num_object_clones': 0, 'num_object_copies': 14764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3693, 'num_write_kb': 29540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3666", 'reported_seq': 3740, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:17.727723+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:17.727723+0000', 'last_peered': '2024-04-26T02:26:17.727723+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:17.727723+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3666, 'log_dups_size': 0, 'ondisk_log_size': 3666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30005510, 'num_objects': 3663, 'num_object_clones': 0, 'num_object_copies': 14652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3665, 'num_write_kb': 29310, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3510", 'reported_seq': 3569, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:13.859169+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:13.859169+0000', 'last_peered': '2024-04-26T02:26:13.859169+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:13.859169+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3510, 'log_dups_size': 0, 'ondisk_log_size': 3510, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28717580, 'num_objects': 3506, 'num_object_clones': 0, 'num_object_copies': 14024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3508, 'num_write_kb': 28053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1538}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3580", 'reported_seq': 3632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.454438+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:15.454438+0000', 'last_peered': '2024-04-26T02:26:15.454438+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:15.454438+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3580, 'log_dups_size': 0, 'ondisk_log_size': 3580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29299212, 'num_objects': 3577, 'num_object_clones': 0, 'num_object_copies': 14308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3578, 'num_write_kb': 28621, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1566}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3694", 'reported_seq': 3733, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.328109+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:15.328109+0000', 'last_peered': '2024-04-26T02:26:15.328109+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:15.328109+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3694, 'log_dups_size': 0, 'ondisk_log_size': 3694, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30261248, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3694, 'num_write_kb': 29552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1643}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3683", 'reported_seq': 6311, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:13.850945+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:26:13.850945+0000', 'last_peered': '2024-04-26T02:26:13.850945+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:26:13.850945+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3683, 'log_dups_size': 0, 'ondisk_log_size': 3683, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30118412, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 802, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3681, 'num_write_kb': 29428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1284, 'num_bytes_recovered': 10485760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2875}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'3807", 'reported_seq': 3868, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:13.710594+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:13.710594+0000', 'last_peered': '2024-04-26T02:26:13.710594+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:13.710594+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3807, 'log_dups_size': 0, 'ondisk_log_size': 3807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31142412, 'num_objects': 3802, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3805, 'num_write_kb': 30428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3653", 'reported_seq': 3705, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.512051+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:15.512051+0000', 'last_peered': '2024-04-26T02:26:15.512051+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:15.512051+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3653, 'log_dups_size': 0, 'ondisk_log_size': 3653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29852696, 'num_objects': 3645, 'num_object_clones': 0, 'num_object_copies': 14580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3650, 'num_write_kb': 29177, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1641}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:20.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3753", 'reported_seq': 3828, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:17.783035+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:17.783035+0000', 'last_peered': '2024-04-26T02:26:17.783035+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:17.783035+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3753, 'log_dups_size': 0, 'ondisk_log_size': 3753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30724620, 'num_objects': 3751, 'num_object_clones': 0, 'num_object_copies': 15004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3751, 'num_write_kb': 30006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1714}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3650", 'reported_seq': 3703, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.188157+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:15.188157+0000', 'last_peered': '2024-04-26T02:26:15.188157+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:15.188157+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3650, 'log_dups_size': 0, 'ondisk_log_size': 3650, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29854482, 'num_objects': 3645, 'num_object_clones': 0, 'num_object_copies': 14580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3647, 'num_write_kb': 29164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3667", 'reported_seq': 3700, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.488216+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:15.488216+0000', 'last_peered': '2024-04-26T02:26:15.488216+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:15.488216+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3667, 'log_dups_size': 0, 'ondisk_log_size': 3667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30030086, 'num_objects': 3666, 'num_object_clones': 0, 'num_object_copies': 14664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3666, 'num_write_kb': 29327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1678}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3764", 'reported_seq': 3835, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:17.619200+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:17.619200+0000', 'last_peered': '2024-04-26T02:26:17.619200+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:17.619200+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3764, 'log_dups_size': 0, 'ondisk_log_size': 3764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30824710, 'num_objects': 3763, 'num_object_clones': 0, 'num_object_copies': 15052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3763, 'num_write_kb': 30103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'3837", 'reported_seq': 3884, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:15.359116+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:15.359116+0000', 'last_peered': '2024-04-26T02:26:15.359116+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:15.359116+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3837, 'log_dups_size': 0, 'ondisk_log_size': 3837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31396364, 'num_objects': 3833, 'num_object_clones': 0, 'num_object_copies': 15332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3835, 'num_write_kb': 30676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:20.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:20.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3784", 'reported_seq': 3856, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:17.757268+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:26:17.757268+0000', 'last_peered': '2024-04-26T02:26:17.757268+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:17.757268+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3784, 'log_dups_size': 0, 'ondisk_log_size': 3784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30970380, 'num_objects': 3781, 'num_object_clones': 0, 'num_object_copies': 15124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3782, 'num_write_kb': 30253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:20.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:20.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3719", 'reported_seq': 3779, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:13.817362+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:13.817362+0000', 'last_peered': '2024-04-26T02:26:13.817362+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:13.817362+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3719, 'log_dups_size': 0, 'ondisk_log_size': 3719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30421516, 'num_objects': 3714, 'num_object_clones': 0, 'num_object_copies': 14856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3717, 'num_write_kb': 29724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:20.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:20.723 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 119 20 60476 60456 3.96748 2.13281 0.0531433 0.0393703 2024-04-26T02:26:21.714 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:21.714 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31348/237964 objects degraded (13.173%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":178,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59491,"data_bytes":487300264,"bytes_used":1639731200,"bytes_avail":384907325440,"bytes_total":386547056640,"degraded_objects":31348,"degraded_total":237964,"degraded_ratio":0.13173421189759796,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":234754,"recovering_keys_per_sec":0,"num_objects_recovered":344,"num_bytes_recovered":2818048,"num_keys_recovered":0,"write_bytes_sec":2426022,"read_op_per_sec":0,"write_op_per_sec":296},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-26T02:25:58.459103+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:21.723 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:26:21.724485+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0394901 2024-04-26T02:26:21.723 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:26:21.723 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 120 20 60789 60769 3.9548 2.44531 0.032838 0.0394901 2024-04-26T02:26:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:21.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:22.532 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:22.546 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-04-26T02:26:22.466390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":492559528,"num_objects":60133,"num_object_clones":0,"num_object_copies":240532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60160,"num_write_kb":481162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":11403264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60187,"ondisk_log_size":60187,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1611144,"kb_used_data":836712,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375876216,"statfs":{"total":386547056640,"available":384897245184,"internally_reserved":0,"allocated":856793088,"data_stored":856089236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":792970803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,13,8,13,7,12,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30588928,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":29872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1851392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004218"},"pg_stats":[{"pgid":"1.f","version":"32'3664","reported_seq":3701,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.486460+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:20.486460+0000","last_peered":"2024-04-26T02:26:20.486460+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:20.486460+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3664,"log_dups_size":0,"ondisk_log_size":3664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":3664,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":29312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3790","reported_seq":3851,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:18.758927+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:18.758927+0000","last_peered":"2024-04-26T02:26:18.758927+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:18.758927+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31011340,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":30300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3666","reported_seq":3740,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.727723+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:17.727723+0000","last_peered":"2024-04-26T02:26:17.727723+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:17.727723+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30005510,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":29310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1639}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3599","reported_seq":3658,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:18.675168+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:18.675168+0000","last_peered":"2024-04-26T02:26:18.675168+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:18.675168+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446668,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3670","reported_seq":3722,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.560085+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:20.560085+0000","last_peered":"2024-04-26T02:26:20.560085+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:20.560085+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3670,"log_dups_size":0,"ondisk_log_size":3670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30036492,"num_objects":3667,"num_object_clones":0,"num_object_copies":14668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3793","reported_seq":3832,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.570330+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:20.570330+0000","last_peered":"2024-04-26T02:26:20.570330+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:20.570330+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31072256,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1742}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3793","reported_seq":6643,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:18.807011+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:18.807011+0000","last_peered":"2024-04-26T02:26:18.807011+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:18.807011+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31019532,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":11403264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'3911","reported_seq":3972,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:18.850399+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:18.850399+0000","last_peered":"2024-04-26T02:26:18.850399+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:18.850399+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31994380,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3739","reported_seq":3791,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.652955+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:20.652955+0000","last_peered":"2024-04-26T02:26:20.652955+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:20.652955+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3739,"log_dups_size":0,"ondisk_log_size":3739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30557208,"num_objects":3731,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3736,"num_write_kb":29865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3753","reported_seq":3828,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.783035+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:17.783035+0000","last_peered":"2024-04-26T02:26:17.783035+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:17.783035+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30724620,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3761","reported_seq":3814,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.547340+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:20.547340+0000","last_peered":"2024-04-26T02:26:20.547340+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:20.547340+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30763794,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3751","reported_seq":3785,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.611032+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:20.611032+0000","last_peered":"2024-04-26T02:26:20.611032+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:20.611032+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30718214,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":29999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3764","reported_seq":3835,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.619200+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:17.619200+0000","last_peered":"2024-04-26T02:26:17.619200+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:17.619200+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30824710,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'3932","reported_seq":3980,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:20.446139+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:20.446139+0000","last_peered":"2024-04-26T02:26:20.446139+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:20.446139+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32174604,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":31436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1857}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3784","reported_seq":3856,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:17.757268+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:17.757268+0000","last_peered":"2024-04-26T02:26:17.757268+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:17.757268+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30970380,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3817","reported_seq":3877,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:18.828105+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:18.828105+0000","last_peered":"2024-04-26T02:26:18.828105+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:18.828105+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31224332,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":30508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":492559528,"num_objects":60133,"num_object_clones":0,"num_object_copies":240532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60160,"num_write_kb":481162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":11403264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":859025408,"data_stored":859025408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60187,"ondisk_log_size":60187,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":453732,"kb_used_data":241564,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93918108,"statfs":{"total":96636764160,"available":96172142592,"internally_reserved":0,"allocated":247361536,"data_stored":247185573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457868,"kb_used_data":237892,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93913972,"statfs":{"total":96636764160,"available":96167907328,"internally_reserved":0,"allocated":243601408,"data_stored":243425445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235424,"kb_used_data":117720,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94136416,"statfs":{"total":96636764160,"available":96395689984,"internally_reserved":0,"allocated":120545280,"data_stored":120369317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":120516303},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464120,"kb_used_data":239536,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93907720,"statfs":{"total":96636764160,"available":96161505280,"internally_reserved":0,"allocated":245284864,"data_stored":245108901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245682176,"data_stored":245682176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121262080,"data_stored":121262080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":244461568,"data_stored":244461568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247619584,"data_stored":247619584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:22.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 121 20 61052 61032 3.93904 2.05469 0.111004 0.0396525 2024-04-26T02:26:23.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 122 20 61383 61363 3.92795 2.58594 0.0411601 0.0397616 2024-04-26T02:26:24.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 123 20 61650 61630 3.91297 2.08594 0.128488 0.0399103 2024-04-26T02:26:25.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:25.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 124 20 61971 61951 3.90164 2.50781 0.0566879 0.0400322 2024-04-26T02:26:26.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 125 20 62274 62254 3.88937 2.36719 0.105142 0.0401546 2024-04-26T02:26:26.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:26.985 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:27.060 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-26T02:26:26.468370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505199784,"num_objects":61676,"num_object_clones":0,"num_object_copies":246704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61703,"num_write_kb":493506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61730,"ondisk_log_size":61730,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1609148,"kb_used_data":855516,"kb_used_omap":14,"kb_used_meta":753585,"kb_avail":375878212,"statfs":{"total":386547056640,"available":384899289088,"internally_reserved":0,"allocated":876048384,"data_stored":875344532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":771671603},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,17,15,10,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30875648,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005142"},"pg_stats":[{"pgid":"1.f","version":"32'3751","reported_seq":3788,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.614819+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:25.614819+0000","last_peered":"2024-04-26T02:26:25.614819+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:25.614819+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30728192,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3913","reported_seq":3974,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.938356+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:24.938356+0000","last_peered":"2024-04-26T02:26:24.938356+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:24.938356+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32018956,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3762","reported_seq":3836,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:22.775140+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:22.775140+0000","last_peered":"2024-04-26T02:26:22.775140+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:22.775140+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30791942,"num_objects":3759,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3761,"num_write_kb":30078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3719","reported_seq":3778,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.912120+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:24.912120+0000","last_peered":"2024-04-26T02:26:24.912120+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:24.912120+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30429708,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3759","reported_seq":3811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.366865+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:25.366865+0000","last_peered":"2024-04-26T02:26:25.366865+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:25.366865+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30765580,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3890","reported_seq":3929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.586435+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:25.586435+0000","last_peered":"2024-04-26T02:26:25.586435+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:25.586435+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31866880,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3890,"num_write_kb":31120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3884","reported_seq":7102,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.980745+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:24.980745+0000","last_peered":"2024-04-26T02:26:24.980745+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:24.980745+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31765004,"num_objects":3878,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4017","reported_seq":4079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.802982+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:24.802982+0000","last_peered":"2024-04-26T02:26:24.802982+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:24.802982+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32862732,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":32108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3850","reported_seq":3903,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.676412+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:25.676412+0000","last_peered":"2024-04-26T02:26:25.676412+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:25.676412+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31466520,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3829","reported_seq":3904,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:22.741129+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:22.741129+0000","last_peered":"2024-04-26T02:26:22.741129+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:22.741129+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347212,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":30614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3872","reported_seq":3926,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.292385+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:25.292385+0000","last_peered":"2024-04-26T02:26:25.292385+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:25.292385+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31673106,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3869,"num_write_kb":30940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3832","reported_seq":3865,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.698227+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:25.698227+0000","last_peered":"2024-04-26T02:26:25.698227+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:25.698227+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31381766,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":30647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3842","reported_seq":3913,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:22.798025+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:22.798025+0000","last_peered":"2024-04-26T02:26:22.798025+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:22.798025+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31463686,"num_objects":3841,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3841,"num_write_kb":30727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1730}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4022","reported_seq":4070,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.618989+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:25.618989+0000","last_peered":"2024-04-26T02:26:25.618989+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:25.618989+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4022,"log_dups_size":0,"ondisk_log_size":4022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911884,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4020,"num_write_kb":32156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1947}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3867","reported_seq":3942,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:22.684484+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:22.684484+0000","last_peered":"2024-04-26T02:26:22.684484+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:22.684484+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31650316,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3921","reported_seq":3982,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.965103+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:24.965103+0000","last_peered":"2024-04-26T02:26:24.965103+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:24.965103+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3921,"log_dups_size":0,"ondisk_log_size":3921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32076300,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":31340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":505199784,"num_objects":61676,"num_object_clones":0,"num_object_copies":246704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61703,"num_write_kb":493506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":878182400,"data_stored":878182400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61730,"ondisk_log_size":61730,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463732,"kb_used_data":247404,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93908108,"statfs":{"total":96636764160,"available":96161902592,"internally_reserved":0,"allocated":253341696,"data_stored":253165733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,5,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451328,"kb_used_data":243832,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93920512,"statfs":{"total":96636764160,"available":96174604288,"internally_reserved":0,"allocated":249683968,"data_stored":249508005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235424,"kb_used_data":117720,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94136416,"statfs":{"total":96636764160,"available":96395689984,"internally_reserved":0,"allocated":120545280,"data_stored":120369317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":120516303},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458664,"kb_used_data":246560,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93913176,"statfs":{"total":96636764160,"available":96167092224,"internally_reserved":0,"allocated":252477440,"data_stored":252301477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,5,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252870656,"data_stored":252870656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121262080,"data_stored":121262080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250228736,"data_stored":250228736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253820928,"data_stored":253820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:27.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:27.725 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 126 20 62579 62559 3.87741 2.38281 0.092966 0.0402793 2024-04-26T02:26:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:28.496 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:28.565 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-04-26T02:26:28.468908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508304552,"num_objects":62055,"num_object_clones":0,"num_object_copies":248220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62082,"num_write_kb":496538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62109,"ondisk_log_size":62109,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1638200,"kb_used_data":867928,"kb_used_omap":14,"kb_used_meta":770225,"kb_avail":375849160,"statfs":{"total":386547056640,"available":384869539840,"internally_reserved":0,"allocated":888758272,"data_stored":888054420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":788710963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,9,12,10,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28499968,"num_objects":3479,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2416640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005152"},"pg_stats":[{"pgid":"1.f","version":"32'3751","reported_seq":3788,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.614819+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:25.614819+0000","last_peered":"2024-04-26T02:26:25.614819+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:25.614819+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30728192,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'3913","reported_seq":3974,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.938356+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:24.938356+0000","last_peered":"2024-04-26T02:26:24.938356+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:24.938356+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32018956,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3857","reported_seq":3931,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.716328+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:27.716328+0000","last_peered":"2024-04-26T02:26:27.716328+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:27.716328+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31570182,"num_objects":3854,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3856,"num_write_kb":30838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3719","reported_seq":3778,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.912120+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:24.912120+0000","last_peered":"2024-04-26T02:26:24.912120+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:24.912120+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30429708,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3759","reported_seq":3811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.366865+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:25.366865+0000","last_peered":"2024-04-26T02:26:25.366865+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:25.366865+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30765580,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3890","reported_seq":3929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.586435+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:25.586435+0000","last_peered":"2024-04-26T02:26:25.586435+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:25.586435+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31866880,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3890,"num_write_kb":31120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3884","reported_seq":7102,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.980745+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:24.980745+0000","last_peered":"2024-04-26T02:26:24.980745+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:24.980745+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31765004,"num_objects":3878,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4017","reported_seq":4079,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.802982+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:24.802982+0000","last_peered":"2024-04-26T02:26:24.802982+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:24.802982+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32862732,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":32108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3850","reported_seq":3903,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.676412+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:25.676412+0000","last_peered":"2024-04-26T02:26:25.676412+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:25.676412+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31466520,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3918","reported_seq":3993,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.832942+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:27.832942+0000","last_peered":"2024-04-26T02:26:27.832942+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:27.832942+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32076300,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3916,"num_write_kb":31326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1879}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3872","reported_seq":3926,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.292385+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:25.292385+0000","last_peered":"2024-04-26T02:26:25.292385+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:25.292385+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31673106,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3869,"num_write_kb":30940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3832","reported_seq":3865,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.698227+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:25.698227+0000","last_peered":"2024-04-26T02:26:25.698227+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:25.698227+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31381766,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":30647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3933","reported_seq":4004,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.763136+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:27.763136+0000","last_peered":"2024-04-26T02:26:27.763136+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:27.763136+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32209158,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1821}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4022","reported_seq":4070,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.618989+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:25.618989+0000","last_peered":"2024-04-26T02:26:25.618989+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:25.618989+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4022,"log_dups_size":0,"ondisk_log_size":4022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911884,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4020,"num_write_kb":32156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1947}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3971","reported_seq":4046,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.753218+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:27.753218+0000","last_peered":"2024-04-26T02:26:27.753218+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:27.753218+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32502284,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":31749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'3921","reported_seq":3982,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:24.965103+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:24.965103+0000","last_peered":"2024-04-26T02:26:24.965103+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:24.965103+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3921,"log_dups_size":0,"ondisk_log_size":3921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32076300,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":31340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508304552,"num_objects":62055,"num_object_clones":0,"num_object_copies":248220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62082,"num_write_kb":496538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":12902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891031552,"data_stored":891031552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62109,"ondisk_log_size":62109,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463732,"kb_used_data":247404,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93908108,"statfs":{"total":96636764160,"available":96161902592,"internally_reserved":0,"allocated":253341696,"data_stored":253165733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,5,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465544,"kb_used_data":249728,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93906296,"statfs":{"total":96636764160,"available":96160047104,"internally_reserved":0,"allocated":255721472,"data_stored":255545509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250260,"kb_used_data":124236,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94121580,"statfs":{"total":96636764160,"available":96380497920,"internally_reserved":0,"allocated":127217664,"data_stored":127041701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458664,"kb_used_data":246560,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93913176,"statfs":{"total":96636764160,"available":96167092224,"internally_reserved":0,"allocated":252477440,"data_stored":252301477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,5,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252870656,"data_stored":252870656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127852544,"data_stored":127852544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256487424,"data_stored":256487424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253820928,"data_stored":253820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3751", 'reported_seq': 3788, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.614819+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:25.614819+0000', 'last_peered': '2024-04-26T02:26:25.614819+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:25.614819+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3751, 'log_dups_size': 0, 'ondisk_log_size': 3751, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30728192, 'num_objects': 3751, 'num_object_clones': 0, 'num_object_copies': 15004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3751, 'num_write_kb': 30008, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1734}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'3913", 'reported_seq': 3974, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:24.938356+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:24.938356+0000', 'last_peered': '2024-04-26T02:26:24.938356+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:24.938356+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3913, 'log_dups_size': 0, 'ondisk_log_size': 3913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32018956, 'num_objects': 3909, 'num_object_clones': 0, 'num_object_copies': 15636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3911, 'num_write_kb': 31284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3857", 'reported_seq': 3931, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:27.716328+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:27.716328+0000', 'last_peered': '2024-04-26T02:26:27.716328+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:27.716328+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3857, 'log_dups_size': 0, 'ondisk_log_size': 3857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31570182, 'num_objects': 3854, 'num_object_clones': 0, 'num_object_copies': 15416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3856, 'num_write_kb': 30838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1830}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3719", 'reported_seq': 3778, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:24.912120+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:24.912120+0000', 'last_peered': '2024-04-26T02:26:24.912120+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:24.912120+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3719, 'log_dups_size': 0, 'ondisk_log_size': 3719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30429708, 'num_objects': 3715, 'num_object_clones': 0, 'num_object_copies': 14860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3717, 'num_write_kb': 29725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:28.567 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3759", 'reported_seq': 3811, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.366865+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:25.366865+0000', 'last_peered': '2024-04-26T02:26:25.366865+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:25.366865+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3759, 'log_dups_size': 0, 'ondisk_log_size': 3759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30765580, 'num_objects': 3756, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3757, 'num_write_kb': 30053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'3890", 'reported_seq': 3929, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.586435+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:25.586435+0000', 'last_peered': '2024-04-26T02:26:25.586435+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:25.586435+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3890, 'log_dups_size': 0, 'ondisk_log_size': 3890, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31866880, 'num_objects': 3890, 'num_object_clones': 0, 'num_object_copies': 15560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3890, 'num_write_kb': 31120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1839}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'3884", 'reported_seq': 7102, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:24.980745+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:26:24.980745+0000', 'last_peered': '2024-04-26T02:26:24.980745+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:26:24.980745+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3884, 'log_dups_size': 0, 'ondisk_log_size': 3884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31765004, 'num_objects': 3878, 'num_object_clones': 0, 'num_object_copies': 15512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 507, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3882, 'num_write_kb': 31036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1579, 'num_bytes_recovered': 12902400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3371}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'4017", 'reported_seq': 4079, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:24.802982+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:24.802982+0000', 'last_peered': '2024-04-26T02:26:24.802982+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:24.802982+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4017, 'log_dups_size': 0, 'ondisk_log_size': 4017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32862732, 'num_objects': 4012, 'num_object_clones': 0, 'num_object_copies': 16048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4015, 'num_write_kb': 32108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'3850", 'reported_seq': 3903, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.676412+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:25.676412+0000', 'last_peered': '2024-04-26T02:26:25.676412+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:25.676412+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3850, 'log_dups_size': 0, 'ondisk_log_size': 3850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31466520, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3847, 'num_write_kb': 30753, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'3918", 'reported_seq': 3993, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:27.832942+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:27.832942+0000', 'last_peered': '2024-04-26T02:26:27.832942+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:27.832942+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3918, 'log_dups_size': 0, 'ondisk_log_size': 3918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32076300, 'num_objects': 3916, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3916, 'num_write_kb': 31326, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1879}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:28.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'3872", 'reported_seq': 3926, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.292385+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:25.292385+0000', 'last_peered': '2024-04-26T02:26:25.292385+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:25.292385+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3872, 'log_dups_size': 0, 'ondisk_log_size': 3872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31673106, 'num_objects': 3867, 'num_object_clones': 0, 'num_object_copies': 15468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3869, 'num_write_kb': 30940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'3832", 'reported_seq': 3865, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.698227+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:25.698227+0000', 'last_peered': '2024-04-26T02:26:25.698227+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:25.698227+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3832, 'log_dups_size': 0, 'ondisk_log_size': 3832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31381766, 'num_objects': 3831, 'num_object_clones': 0, 'num_object_copies': 15324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3831, 'num_write_kb': 30647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'3933", 'reported_seq': 4004, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:27.763136+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:27.763136+0000', 'last_peered': '2024-04-26T02:26:27.763136+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:27.763136+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3933, 'log_dups_size': 0, 'ondisk_log_size': 3933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32209158, 'num_objects': 3932, 'num_object_clones': 0, 'num_object_copies': 15728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3932, 'num_write_kb': 31455, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'4022", 'reported_seq': 4070, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:25.618989+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:25.618989+0000', 'last_peered': '2024-04-26T02:26:25.618989+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:25.618989+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4022, 'log_dups_size': 0, 'ondisk_log_size': 4022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32911884, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4020, 'num_write_kb': 32156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1947}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'3971", 'reported_seq': 4046, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:27.753218+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:26:27.753218+0000', 'last_peered': '2024-04-26T02:26:27.753218+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:27.753218+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3971, 'log_dups_size': 0, 'ondisk_log_size': 3971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32502284, 'num_objects': 3968, 'num_object_clones': 0, 'num_object_copies': 15872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3969, 'num_write_kb': 31749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:28.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'3921", 'reported_seq': 3982, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:24.965103+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:24.965103+0000', 'last_peered': '2024-04-26T02:26:24.965103+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:24.965103+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3921, 'log_dups_size': 0, 'ondisk_log_size': 3921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32076300, 'num_objects': 3916, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3919, 'num_write_kb': 31340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1852}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:28.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:28.727 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 127 20 62871 62851 3.86484 2.28125 0.0218956 0.0404062 2024-04-26T02:26:29.733 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 128 20 63170 63150 3.8528 2.33594 0.0202139 0.0405337 2024-04-26T02:26:30.253 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:30.253 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31165/248220 objects degraded (12.555%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":186,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62055,"data_bytes":508304552,"bytes_used":1677516800,"bytes_avail":384869539840,"bytes_total":386547056640,"degraded_objects":31165,"degraded_total":248220,"degraded_ratio":0.12555394408186285,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":201300,"recovering_keys_per_sec":0,"num_objects_recovered":295,"num_bytes_recovered":2416640,"num_keys_recovered":0,"write_bytes_sec":2373978,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-26T02:25:58.459103+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:30.303 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:30.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:30.730 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 129 20 63456 63436 3.84026 2.23438 0.0657633 0.0406682 2024-04-26T02:26:31.109 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:31.109 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:31.229 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-26T02:26:30.470548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512318632,"num_objects":62545,"num_object_clones":0,"num_object_copies":250180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62572,"num_write_kb":500458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62599,"ondisk_log_size":62599,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1648476,"kb_used_data":874044,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375838884,"statfs":{"total":386547056640,"available":384859017216,"internally_reserved":0,"allocated":895021056,"data_stored":894317204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":792970803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,7,11,13,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29081600,"num_objects":3550,"num_object_clones":0,"num_object_copies":14200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3550,"num_write_kb":28400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":3186688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005198"},"pg_stats":[{"pgid":"1.f","version":"32'3751","reported_seq":3788,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.614819+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:25.614819+0000","last_peered":"2024-04-26T02:26:25.614819+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:25.614819+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30728192,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4003","reported_seq":4064,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.926294+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:29.926294+0000","last_peered":"2024-04-26T02:26:29.926294+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:29.926294+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756236,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":32004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3857","reported_seq":3931,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.716328+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:27.716328+0000","last_peered":"2024-04-26T02:26:27.716328+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:27.716328+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31570182,"num_objects":3854,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3856,"num_write_kb":30838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3824","reported_seq":3883,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.923237+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:29.923237+0000","last_peered":"2024-04-26T02:26:29.923237+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:29.923237+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31289868,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3822,"num_write_kb":30565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3759","reported_seq":3811,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.366865+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:25.366865+0000","last_peered":"2024-04-26T02:26:25.366865+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:25.366865+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30765580,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3757,"num_write_kb":30053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3890","reported_seq":3929,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.586435+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:25.586435+0000","last_peered":"2024-04-26T02:26:25.586435+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:25.586435+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31866880,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3890,"num_write_kb":31120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3993","reported_seq":7397,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.982231+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:29.982231+0000","last_peered":"2024-04-26T02:26:29.982231+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:29.982231+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32657932,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3991,"num_write_kb":31908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1673,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4110","reported_seq":4172,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.900867+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:29.900867+0000","last_peered":"2024-04-26T02:26:29.900867+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:29.900867+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33624588,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3850","reported_seq":3903,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.676412+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:25.676412+0000","last_peered":"2024-04-26T02:26:25.676412+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:25.676412+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31466520,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'3918","reported_seq":3993,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.832942+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:27.832942+0000","last_peered":"2024-04-26T02:26:27.832942+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:27.832942+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32076300,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3916,"num_write_kb":31326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1879}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3872","reported_seq":3926,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.292385+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:25.292385+0000","last_peered":"2024-04-26T02:26:25.292385+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:25.292385+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31673106,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3869,"num_write_kb":30940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3832","reported_seq":3865,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.698227+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:25.698227+0000","last_peered":"2024-04-26T02:26:25.698227+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:25.698227+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31381766,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":30647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'3933","reported_seq":4004,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.763136+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:27.763136+0000","last_peered":"2024-04-26T02:26:27.763136+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:27.763136+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32209158,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1821}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4022","reported_seq":4070,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:25.618989+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:25.618989+0000","last_peered":"2024-04-26T02:26:25.618989+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:25.618989+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4022,"log_dups_size":0,"ondisk_log_size":4022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32911884,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4020,"num_write_kb":32156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1947}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'3971","reported_seq":4046,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:27.753218+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:27.753218+0000","last_peered":"2024-04-26T02:26:27.753218+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:27.753218+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32502284,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":31749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1915}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4014","reported_seq":4075,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.948624+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:29.948624+0000","last_peered":"2024-04-26T02:26:29.948624+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:29.948624+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32838156,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1945}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":512318632,"num_objects":62545,"num_object_clones":0,"num_object_copies":250180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62572,"num_write_kb":500458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":897277952,"data_stored":897277952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62599,"ondisk_log_size":62599,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463732,"kb_used_data":247404,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93908108,"statfs":{"total":96636764160,"available":96161902592,"internally_reserved":0,"allocated":253341696,"data_stored":253165733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,5,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465544,"kb_used_data":249728,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93906296,"statfs":{"total":96636764160,"available":96160047104,"internally_reserved":0,"allocated":255721472,"data_stored":255545509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250260,"kb_used_data":124236,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94121580,"statfs":{"total":96636764160,"available":96380497920,"internally_reserved":0,"allocated":127217664,"data_stored":127041701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468940,"kb_used_data":252676,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93902900,"statfs":{"total":96636764160,"available":96156569600,"internally_reserved":0,"allocated":258740224,"data_stored":258564261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259117056,"data_stored":259117056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127852544,"data_stored":127852544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256487424,"data_stored":256487424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253820928,"data_stored":253820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:31.730 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 130 20 63765 63745 3.82927 2.41406 0.0252497 0.0407777 2024-04-26T02:26:32.730 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 131 20 64074 64054 3.81847 2.41406 0.0867171 0.0409027 2024-04-26T02:26:33.730 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 132 20 64390 64370 3.80825 2.46875 0.068051 0.0410094 2024-04-26T02:26:34.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:34.733 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 133 19 64686 64667 3.79706 2.32031 0.0562658 0.0411296 2024-04-26T02:26:35.580 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:35.580 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:35.614 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-04-26T02:26:34.471633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":520838312,"num_objects":63585,"num_object_clones":0,"num_object_copies":254340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63612,"num_write_kb":508778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63639,"ondisk_log_size":63639,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1697668,"kb_used_data":892708,"kb_used_omap":14,"kb_used_meta":804913,"kb_avail":375789692,"statfs":{"total":386547056640,"available":384808644608,"internally_reserved":0,"allocated":914132992,"data_stored":913429140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":824231475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,9,11,14,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28278784,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":27616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2269184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005243"},"pg_stats":[{"pgid":"1.f","version":"32'3834","reported_seq":3871,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.838793+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:30.838793+0000","last_peered":"2024-04-26T02:26:30.838793+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:30.838793+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3834,"log_dups_size":0,"ondisk_log_size":3834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31408128,"num_objects":3834,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3834,"num_write_kb":30672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4003","reported_seq":4064,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.926294+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:29.926294+0000","last_peered":"2024-04-26T02:26:29.926294+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:29.926294+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32756236,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":32004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3937","reported_seq":4011,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.830822+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:32.830822+0000","last_peered":"2024-04-26T02:26:32.830822+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:32.830822+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225542,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3824","reported_seq":3883,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.923237+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:29.923237+0000","last_peered":"2024-04-26T02:26:29.923237+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:29.923237+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31289868,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3822,"num_write_kb":30565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3856","reported_seq":3908,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.696830+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:30.696830+0000","last_peered":"2024-04-26T02:26:30.696830+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:30.696830+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560204,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3854,"num_write_kb":30829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'3975","reported_seq":4014,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.779140+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:30.779140+0000","last_peered":"2024-04-26T02:26:30.779140+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:30.779140+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'3993","reported_seq":7397,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.982231+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:29.982231+0000","last_peered":"2024-04-26T02:26:29.982231+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:29.982231+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32657932,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3991,"num_write_kb":31908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1673,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4110","reported_seq":4172,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.900867+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:29.900867+0000","last_peered":"2024-04-26T02:26:29.900867+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:29.900867+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33624588,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'3954","reported_seq":4007,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.812829+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:30.812829+0000","last_peered":"2024-04-26T02:26:30.812829+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:30.812829+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32318488,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3951,"num_write_kb":31585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1942}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4013","reported_seq":4088,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.758242+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:32.758242+0000","last_peered":"2024-04-26T02:26:32.758242+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:32.758242+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32854540,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4011,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1974}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'3989","reported_seq":4043,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.795179+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:30.795179+0000","last_peered":"2024-04-26T02:26:30.795179+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:30.795179+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32631570,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":31876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'3929","reported_seq":3964,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.805816+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:30.805816+0000","last_peered":"2024-04-26T02:26:30.805816+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:30.805816+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3929,"log_dups_size":0,"ondisk_log_size":3929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32176390,"num_objects":3928,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3928,"num_write_kb":31423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4038","reported_seq":4109,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.830543+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:32.830543+0000","last_peered":"2024-04-26T02:26:32.830543+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:32.830543+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33069318,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":32295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4109","reported_seq":4156,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:30.816011+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:30.816011+0000","last_peered":"2024-04-26T02:26:30.816011+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:30.816011+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33624588,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2034}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4061","reported_seq":4136,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.793090+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:32.793090+0000","last_peered":"2024-04-26T02:26:32.793090+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:32.793090+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4061,"log_dups_size":0,"ondisk_log_size":4061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33239564,"num_objects":4058,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":32469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4014","reported_seq":4075,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:29.948624+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:29.948624+0000","last_peered":"2024-04-26T02:26:29.948624+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:29.948624+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32838156,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1945}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":520838312,"num_objects":63585,"num_object_clones":0,"num_object_copies":254340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63612,"num_write_kb":508778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1698,"num_bytes_recovered":13672448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":916226048,"data_stored":916226048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63639,"ondisk_log_size":63639,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474024,"kb_used_data":253536,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93897816,"statfs":{"total":96636764160,"available":96151363584,"internally_reserved":0,"allocated":259620864,"data_stored":259444901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,5,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475812,"kb_used_data":255836,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93896028,"statfs":{"total":96636764160,"available":96149532672,"internally_reserved":0,"allocated":261976064,"data_stored":261800101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278892,"kb_used_data":130660,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94092948,"statfs":{"total":96636764160,"available":96351178752,"internally_reserved":0,"allocated":133795840,"data_stored":133619877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468940,"kb_used_data":252676,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93902900,"statfs":{"total":96636764160,"available":96156569600,"internally_reserved":0,"allocated":258740224,"data_stored":258564261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259117056,"data_stored":259117056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134438912,"data_stored":134438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":262606848,"data_stored":262606848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":260063232,"data_stored":260063232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:35.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:35.730 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 134 20 64952 64932 3.78417 2.07031 0.210866 0.0412369 2024-04-26T02:26:36.732 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 135 20 65266 65246 3.77428 2.45312 0.0466057 0.0413827 2024-04-26T02:26:36.763 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:36.764 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:36.854 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-04-26T02:26:36.472153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530283688,"num_objects":64738,"num_object_clones":0,"num_object_copies":258952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64765,"num_write_kb":518002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64792,"ondisk_log_size":64792,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1693152,"kb_used_data":904832,"kb_used_omap":14,"kb_used_meta":788273,"kb_avail":375794208,"statfs":{"total":386547056640,"available":384813268992,"internally_reserved":0,"allocated":926547968,"data_stored":925844116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":807192115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,9,2,7,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34996224,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":34176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":3211264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005333"},"pg_stats":[{"pgid":"1.f","version":"32'3929","reported_seq":3967,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.864442+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:35.864442+0000","last_peered":"2024-04-26T02:26:35.864442+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:35.864442+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3929,"log_dups_size":0,"ondisk_log_size":3929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32186368,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3929,"num_write_kb":31432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1912}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4093","reported_seq":4155,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.952882+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:34.952882+0000","last_peered":"2024-04-26T02:26:34.952882+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:34.952882+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33493516,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":32724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'3937","reported_seq":4011,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.830822+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:32.830822+0000","last_peered":"2024-04-26T02:26:32.830822+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:32.830822+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32225542,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3911","reported_seq":3970,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.938660+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:34.938660+0000","last_peered":"2024-04-26T02:26:34.938660+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:34.938660+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32002572,"num_objects":3907,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3949","reported_seq":4001,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.854243+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:35.854243+0000","last_peered":"2024-04-26T02:26:35.854243+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:35.854243+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32322060,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3947,"num_write_kb":31573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'4083","reported_seq":4123,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.809534+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:35.809534+0000","last_peered":"2024-04-26T02:26:35.809534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:35.809534+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33447936,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'4095","reported_seq":7731,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.035886+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:35.035886+0000","last_peered":"2024-04-26T02:26:35.035886+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:35.035886+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33493516,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":32724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1788,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4221","reported_seq":4283,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.925097+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:34.925097+0000","last_peered":"2024-04-26T02:26:34.925097+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:34.925097+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533900,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":33740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'4067","reported_seq":4120,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.792285+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:35.792285+0000","last_peered":"2024-04-26T02:26:35.792285+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:35.792285+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33244184,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4013","reported_seq":4088,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.758242+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:32.758242+0000","last_peered":"2024-04-26T02:26:32.758242+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:32.758242+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32854540,"num_objects":4011,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4011,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1974}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'4073","reported_seq":4127,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.844752+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:35.844752+0000","last_peered":"2024-04-26T02:26:35.844752+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:35.844752+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33319698,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":32548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'4028","reported_seq":4062,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.912508+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:35.912508+0000","last_peered":"2024-04-26T02:26:35.912508+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:35.912508+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987398,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":32215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4038","reported_seq":4109,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.830543+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:32.830543+0000","last_peered":"2024-04-26T02:26:32.830543+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:32.830543+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33069318,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":32295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4197","reported_seq":4246,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.851661+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:35.851661+0000","last_peered":"2024-04-26T02:26:35.851661+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:35.851661+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34345484,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4061","reported_seq":4136,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:32.793090+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:32.793090+0000","last_peered":"2024-04-26T02:26:32.793090+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:32.793090+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4061,"log_dups_size":0,"ondisk_log_size":4061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33239564,"num_objects":4058,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":32469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4097","reported_seq":4158,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.998807+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:34.998807+0000","last_peered":"2024-04-26T02:26:34.998807+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:34.998807+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33518092,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530283688,"num_objects":64738,"num_object_clones":0,"num_object_copies":258952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64765,"num_write_kb":518002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":928702464,"data_stored":928702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64792,"ondisk_log_size":64792,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471632,"kb_used_data":259464,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900208,"statfs":{"total":96636764160,"available":96153812992,"internally_reserved":0,"allocated":265691136,"data_stored":265515173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475812,"kb_used_data":255836,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93896028,"statfs":{"total":96636764160,"available":96149532672,"internally_reserved":0,"allocated":261976064,"data_stored":261800101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278892,"kb_used_data":130660,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94092948,"statfs":{"total":96636764160,"available":96351178752,"internally_reserved":0,"allocated":133795840,"data_stored":133619877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151776975},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466816,"kb_used_data":258872,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93905024,"statfs":{"total":96636764160,"available":96158744576,"internally_reserved":0,"allocated":265084928,"data_stored":264908965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212922063},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265347072,"data_stored":265347072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134438912,"data_stored":134438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":262606848,"data_stored":262606848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266309632,"data_stored":266309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:36.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:36.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'3929", 'reported_seq': 3967, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.864442+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:35.864442+0000', 'last_peered': '2024-04-26T02:26:35.864442+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:35.864442+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3929, 'log_dups_size': 0, 'ondisk_log_size': 3929, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32186368, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3929, 'num_write_kb': 31432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1912}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:36.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'4093", 'reported_seq': 4155, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:34.952882+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:34.952882+0000', 'last_peered': '2024-04-26T02:26:34.952882+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:34.952882+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4093, 'log_dups_size': 0, 'ondisk_log_size': 4093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33493516, 'num_objects': 4089, 'num_object_clones': 0, 'num_object_copies': 16356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4091, 'num_write_kb': 32724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'3937", 'reported_seq': 4011, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:32.830822+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:32.830822+0000', 'last_peered': '2024-04-26T02:26:32.830822+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:32.830822+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3937, 'log_dups_size': 0, 'ondisk_log_size': 3937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32225542, 'num_objects': 3934, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3936, 'num_write_kb': 31478, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1910}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3911", 'reported_seq': 3970, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:34.938660+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:34.938660+0000', 'last_peered': '2024-04-26T02:26:34.938660+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:34.938660+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3911, 'log_dups_size': 0, 'ondisk_log_size': 3911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32002572, 'num_objects': 3907, 'num_object_clones': 0, 'num_object_copies': 15628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3909, 'num_write_kb': 31261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'3949", 'reported_seq': 4001, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.854243+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:35.854243+0000', 'last_peered': '2024-04-26T02:26:35.854243+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:35.854243+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3949, 'log_dups_size': 0, 'ondisk_log_size': 3949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32322060, 'num_objects': 3946, 'num_object_clones': 0, 'num_object_copies': 15784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3947, 'num_write_kb': 31573, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'4083", 'reported_seq': 4123, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.809534+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:35.809534+0000', 'last_peered': '2024-04-26T02:26:35.809534+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:35.809534+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4083, 'log_dups_size': 0, 'ondisk_log_size': 4083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33447936, 'num_objects': 4083, 'num_object_clones': 0, 'num_object_copies': 16332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4083, 'num_write_kb': 32664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2032}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'4095", 'reported_seq': 7731, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.035886+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:26:35.035886+0000', 'last_peered': '2024-04-26T02:26:35.035886+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:26:35.035886+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4095, 'log_dups_size': 0, 'ondisk_log_size': 4095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33493516, 'num_objects': 4089, 'num_object_clones': 0, 'num_object_copies': 16356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4093, 'num_write_kb': 32724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1788, 'num_bytes_recovered': 14614528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3791}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:36.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'4221", 'reported_seq': 4283, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:34.925097+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:34.925097+0000', 'last_peered': '2024-04-26T02:26:34.925097+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:34.925097+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4221, 'log_dups_size': 0, 'ondisk_log_size': 4221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34533900, 'num_objects': 4216, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4219, 'num_write_kb': 33740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'4067", 'reported_seq': 4120, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.792285+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:35.792285+0000', 'last_peered': '2024-04-26T02:26:35.792285+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:35.792285+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4067, 'log_dups_size': 0, 'ondisk_log_size': 4067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33244184, 'num_objects': 4059, 'num_object_clones': 0, 'num_object_copies': 16236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4064, 'num_write_kb': 32489, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2055}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'4013", 'reported_seq': 4088, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:32.758242+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:32.758242+0000', 'last_peered': '2024-04-26T02:26:32.758242+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:32.758242+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4013, 'log_dups_size': 0, 'ondisk_log_size': 4013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32854540, 'num_objects': 4011, 'num_object_clones': 0, 'num_object_copies': 16044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4011, 'num_write_kb': 32086, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1974}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'4073", 'reported_seq': 4127, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.844752+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:35.844752+0000', 'last_peered': '2024-04-26T02:26:35.844752+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:35.844752+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4073, 'log_dups_size': 0, 'ondisk_log_size': 4073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33319698, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4070, 'num_write_kb': 32548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2051}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'4028", 'reported_seq': 4062, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.912508+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:35.912508+0000', 'last_peered': '2024-04-26T02:26:35.912508+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:35.912508+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4028, 'log_dups_size': 0, 'ondisk_log_size': 4028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32987398, 'num_objects': 4027, 'num_object_clones': 0, 'num_object_copies': 16108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4027, 'num_write_kb': 32215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:36.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'4038", 'reported_seq': 4109, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:32.830543+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:32.830543+0000', 'last_peered': '2024-04-26T02:26:32.830543+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:32.830543+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4038, 'log_dups_size': 0, 'ondisk_log_size': 4038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33069318, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 16148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 32295, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1926}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'4197", 'reported_seq': 4246, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:35.851661+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:35.851661+0000', 'last_peered': '2024-04-26T02:26:35.851661+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:35.851661+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34345484, 'num_objects': 4193, 'num_object_clones': 0, 'num_object_copies': 16772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4195, 'num_write_kb': 33556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'4061", 'reported_seq': 4136, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:32.793090+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:26:32.793090+0000', 'last_peered': '2024-04-26T02:26:32.793090+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:32.793090+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4061, 'log_dups_size': 0, 'ondisk_log_size': 4061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33239564, 'num_objects': 4058, 'num_object_clones': 0, 'num_object_copies': 16232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4059, 'num_write_kb': 32469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:36.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'4097", 'reported_seq': 4158, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:34.998807+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:34.998807+0000', 'last_peered': '2024-04-26T02:26:34.998807+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:34.998807+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4097, 'log_dups_size': 0, 'ondisk_log_size': 4097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33518092, 'num_objects': 4092, 'num_object_clones': 0, 'num_object_copies': 16368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4095, 'num_write_kb': 32748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:36.858 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:37.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 136 20 65563 65543 3.76355 2.32031 0.0382368 0.0414992 2024-04-26T02:26:38.647 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:38.647 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30956/258952 objects degraded (11.954%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":195,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64738,"data_bytes":530283688,"bytes_used":1733787648,"bytes_avail":384813268992,"bytes_total":386547056640,"degraded_objects":30956,"degraded_total":258952,"degraded_ratio":0.11954339028082424,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":267486,"recovering_keys_per_sec":0,"num_objects_recovered":392,"num_bytes_recovered":3211264,"num_keys_recovered":0,"write_bytes_sec":2915056,"read_op_per_sec":0,"write_op_per_sec":355},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:38.733 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 137 20 65869 65849 3.75353 2.39062 0.028488 0.0416099 2024-04-26T02:26:38.777 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:38.777 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:39.733 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 138 20 66153 66133 3.74241 2.21875 0.0202674 0.0417338 2024-04-26T02:26:39.856 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:39.857 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:39.895 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-04-26T02:26:38.472727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":533257384,"num_objects":65101,"num_object_clones":0,"num_object_copies":260404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65128,"num_write_kb":520906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65155,"ondisk_log_size":65155,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1702964,"kb_used_data":917396,"kb_used_omap":14,"kb_used_meta":785521,"kb_avail":375784396,"statfs":{"total":386547056640,"available":384803221504,"internally_reserved":0,"allocated":939413504,"data_stored":938709652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":804374067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,6,1,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28057600,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3425,"num_write_kb":27400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004356"},"pg_stats":[{"pgid":"1.f","version":"32'3929","reported_seq":3967,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.864442+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:35.864442+0000","last_peered":"2024-04-26T02:26:35.864442+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:35.864442+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3929,"log_dups_size":0,"ondisk_log_size":3929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32186368,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3929,"num_write_kb":31432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1912}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4093","reported_seq":4155,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.952882+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:34.952882+0000","last_peered":"2024-04-26T02:26:34.952882+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:34.952882+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33493516,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":32724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'4037","reported_seq":4111,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.691560+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:37.691560+0000","last_peered":"2024-04-26T02:26:37.691560+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:37.691560+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33044742,"num_objects":4034,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":32278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3911","reported_seq":3970,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.938660+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:34.938660+0000","last_peered":"2024-04-26T02:26:34.938660+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:34.938660+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32002572,"num_objects":3907,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'3949","reported_seq":4001,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.854243+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:35.854243+0000","last_peered":"2024-04-26T02:26:35.854243+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:35.854243+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32322060,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3947,"num_write_kb":31573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'4083","reported_seq":4123,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.809534+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:35.809534+0000","last_peered":"2024-04-26T02:26:35.809534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:35.809534+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33447936,"num_objects":4083,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'4095","reported_seq":7731,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.035886+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:35.035886+0000","last_peered":"2024-04-26T02:26:35.035886+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:35.035886+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33493516,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":32724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1788,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4221","reported_seq":4283,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.925097+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:34.925097+0000","last_peered":"2024-04-26T02:26:34.925097+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:34.925097+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34533900,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":33740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'4067","reported_seq":4120,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.792285+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:35.792285+0000","last_peered":"2024-04-26T02:26:35.792285+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:35.792285+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33244184,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4111","reported_seq":4186,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.602500+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:37.602500+0000","last_peered":"2024-04-26T02:26:37.602500+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:37.602500+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33657356,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":32870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2072}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'4073","reported_seq":4127,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.844752+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:35.844752+0000","last_peered":"2024-04-26T02:26:35.844752+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:35.844752+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33319698,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":32548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'4028","reported_seq":4062,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.912508+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:35.912508+0000","last_peered":"2024-04-26T02:26:35.912508+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:35.912508+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987398,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":32215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2039}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4125","reported_seq":4196,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.865747+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:37.865747+0000","last_peered":"2024-04-26T02:26:37.865747+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:37.865747+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33782022,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4197","reported_seq":4246,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:35.851661+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:35.851661+0000","last_peered":"2024-04-26T02:26:35.851661+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:35.851661+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34345484,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4139","reported_seq":4214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.761906+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:37.761906+0000","last_peered":"2024-04-26T02:26:37.761906+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:37.761906+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33878540,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":33093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4097","reported_seq":4158,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:34.998807+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:34.998807+0000","last_peered":"2024-04-26T02:26:34.998807+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:34.998807+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33518092,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":533257384,"num_objects":65101,"num_object_clones":0,"num_object_copies":260404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65128,"num_write_kb":520906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":941559808,"data_stored":941559808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65155,"ondisk_log_size":65155,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471632,"kb_used_data":259464,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900208,"statfs":{"total":96636764160,"available":96153812992,"internally_reserved":0,"allocated":265691136,"data_stored":265515173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469384,"kb_used_data":261888,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93902456,"statfs":{"total":96636764160,"available":96156114944,"internally_reserved":0,"allocated":268173312,"data_stored":267997349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295132,"kb_used_data":137172,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94076708,"statfs":{"total":96636764160,"available":96334548992,"internally_reserved":0,"allocated":140464128,"data_stored":140288165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466816,"kb_used_data":258872,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93905024,"statfs":{"total":96636764160,"available":96158744576,"internally_reserved":0,"allocated":265084928,"data_stored":264908965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212922063},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265347072,"data_stored":265347072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141160448,"data_stored":141160448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268742656,"data_stored":268742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266309632,"data_stored":266309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:40.734 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 139 20 66467 66447 3.73314 2.45312 0.126587 0.0418358 2024-04-26T02:26:41.734 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:26:41.735887+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0419339 2024-04-26T02:26:41.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:26:41.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 140 20 66778 66758 3.72381 2.42969 0.0349889 0.0419339 2024-04-26T02:26:42.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 141 20 67075 67055 3.71385 2.32031 0.0525511 0.042051 2024-04-26T02:26:42.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:43.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 142 20 67382 67362 3.70459 2.39844 0.0430768 0.0421616 2024-04-26T02:26:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:44.392 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:44.545 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-04-26T02:26:42.473655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542457000,"num_objects":66224,"num_object_clones":0,"num_object_copies":264896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66251,"num_write_kb":529890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66278,"ondisk_log_size":66278,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1733660,"kb_used_data":935612,"kb_used_omap":14,"kb_used_meta":798001,"kb_avail":375753700,"statfs":{"total":386547056640,"available":384771788800,"internally_reserved":0,"allocated":958066688,"data_stored":957362836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":817153587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,11,2,4,6,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30138368,"num_objects":3679,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":29432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"1.f","version":"32'4035","reported_seq":4073,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.872317+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:40.872317+0000","last_peered":"2024-04-26T02:26:40.872317+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:40.872317+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":32280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4179","reported_seq":4241,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:39.959406+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:39.959406+0000","last_peered":"2024-04-26T02:26:39.959406+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:39.959406+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34198028,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":33412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'4037","reported_seq":4111,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.691560+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:37.691560+0000","last_peered":"2024-04-26T02:26:37.691560+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:37.691560+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33044742,"num_objects":4034,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":32278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3990","reported_seq":4049,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:39.994664+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:39.994664+0000","last_peered":"2024-04-26T02:26:39.994664+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:39.994664+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3990,"log_dups_size":0,"ondisk_log_size":3990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32649740,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'4030","reported_seq":4082,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.802036+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:40.802036+0000","last_peered":"2024-04-26T02:26:40.802036+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:40.802036+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32985612,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2016}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'4172","reported_seq":4212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.907390+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:40.907390+0000","last_peered":"2024-04-26T02:26:40.907390+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:40.907390+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34177024,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":33376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'4194","reported_seq":8148,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.029556+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:40.029556+0000","last_peered":"2024-04-26T02:26:40.029556+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:40.029556+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34304524,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4323","reported_seq":4384,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.043664+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:40.043664+0000","last_peered":"2024-04-26T02:26:40.043664+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:40.043664+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35369484,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'4168","reported_seq":4221,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.923911+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:40.923911+0000","last_peered":"2024-04-26T02:26:40.923911+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:40.923911+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34071576,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4165,"num_write_kb":33297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4111","reported_seq":4186,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.602500+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:37.602500+0000","last_peered":"2024-04-26T02:26:37.602500+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:37.602500+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33657356,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":32870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2072}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'4173","reported_seq":4227,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.931063+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:40.931063+0000","last_peered":"2024-04-26T02:26:40.931063+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:40.931063+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34138898,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":33348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'4124","reported_seq":4159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.954214+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:40.954214+0000","last_peered":"2024-04-26T02:26:40.954214+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:40.954214+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33773830,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4123,"num_write_kb":32983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4125","reported_seq":4196,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.865747+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:37.865747+0000","last_peered":"2024-04-26T02:26:37.865747+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:37.865747+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33782022,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4296","reported_seq":4345,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.901749+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:40.901749+0000","last_peered":"2024-04-26T02:26:40.901749+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:40.901749+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35156492,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":34348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4139","reported_seq":4214,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:37.761906+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:37.761906+0000","last_peered":"2024-04-26T02:26:37.761906+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:37.761906+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33878540,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":33093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4182","reported_seq":4243,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.006638+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:40.006638+0000","last_peered":"2024-04-26T02:26:40.006638+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:40.006638+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34214412,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":33428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542457000,"num_objects":66224,"num_object_clones":0,"num_object_copies":264896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66251,"num_write_kb":529890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":960593920,"data_stored":960593920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66278,"ondisk_log_size":66278,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481516,"kb_used_data":265188,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93890324,"statfs":{"total":96636764160,"available":96143691776,"internally_reserved":0,"allocated":271552512,"data_stored":271376549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1,5,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469384,"kb_used_data":261888,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93902456,"statfs":{"total":96636764160,"available":96156114944,"internally_reserved":0,"allocated":268173312,"data_stored":267997349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305844,"kb_used_data":143724,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94065996,"statfs":{"total":96636764160,"available":96323579904,"internally_reserved":0,"allocated":147173376,"data_stored":146997413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476916,"kb_used_data":264812,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93894924,"statfs":{"total":96636764160,"available":96148402176,"internally_reserved":0,"allocated":271167488,"data_stored":270991525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271335424,"data_stored":271335424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148004864,"data_stored":148004864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268742656,"data_stored":268742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272510976,"data_stored":272510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:44.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:44.737 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 143 20 67686 67666 3.69529 2.375 0.0596094 0.0422668 2024-04-26T02:26:45.616 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:45.617 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:45.645 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-04-26T02:26:44.754516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545406120,"num_objects":66584,"num_object_clones":0,"num_object_copies":266336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66611,"num_write_kb":532770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66638,"ondisk_log_size":66638,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1748104,"kb_used_data":941736,"kb_used_omap":14,"kb_used_meta":806321,"kb_avail":375739256,"statfs":{"total":386547056640,"available":384756998144,"internally_reserved":0,"allocated":964337664,"data_stored":963633812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":825673267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,11,15,6,4,6,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24567808,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":23992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.282882"},"pg_stats":[{"pgid":"1.f","version":"32'4035","reported_seq":4073,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.872317+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:40.872317+0000","last_peered":"2024-04-26T02:26:40.872317+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:40.872317+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":32280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'4179","reported_seq":4241,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:39.959406+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:39.959406+0000","last_peered":"2024-04-26T02:26:39.959406+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:39.959406+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34198028,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":33412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'4121","reported_seq":4195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.882162+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:42.882162+0000","last_peered":"2024-04-26T02:26:42.882162+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:42.882162+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33732870,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4120,"num_write_kb":32950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'3990","reported_seq":4049,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:39.994664+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:39.994664+0000","last_peered":"2024-04-26T02:26:39.994664+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:39.994664+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":3990,"log_dups_size":0,"ondisk_log_size":3990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32649740,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"32'4030","reported_seq":4082,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.802036+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:40.802036+0000","last_peered":"2024-04-26T02:26:40.802036+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:40.802036+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32985612,"num_objects":4027,"num_object_clones":0,"num_object_copies":16108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":32221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2016}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"32'4172","reported_seq":4212,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.907390+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:40.907390+0000","last_peered":"2024-04-26T02:26:40.907390+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:40.907390+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34177024,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":33376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'4194","reported_seq":8148,"reported_epoch":32,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.029556+0000","last_change":"2024-04-26T02:24:59.527447+0000","last_active":"2024-04-26T02:26:40.029556+0000","last_peered":"2024-04-26T02:26:40.029556+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:40.029556+0000","last_undegraded":"2024-04-26T02:24:59.264738+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34304524,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'4323","reported_seq":4384,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.043664+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:40.043664+0000","last_peered":"2024-04-26T02:26:40.043664+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:40.043664+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35369484,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'4168","reported_seq":4221,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.923911+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:40.923911+0000","last_peered":"2024-04-26T02:26:40.923911+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:40.923911+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34071576,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4165,"num_write_kb":33297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4199","reported_seq":4274,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.859992+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:42.859992+0000","last_peered":"2024-04-26T02:26:42.859992+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:42.859992+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34378252,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":33574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"32'4173","reported_seq":4227,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.931063+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:40.931063+0000","last_peered":"2024-04-26T02:26:40.931063+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:40.931063+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34138898,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":33348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"32'4124","reported_seq":4159,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.954214+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:40.954214+0000","last_peered":"2024-04-26T02:26:40.954214+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:40.954214+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33773830,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4123,"num_write_kb":32983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4215","reported_seq":4286,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.782850+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:42.782850+0000","last_peered":"2024-04-26T02:26:42.782850+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:42.782850+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34519302,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":33711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"32'4296","reported_seq":4345,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.901749+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:40.901749+0000","last_peered":"2024-04-26T02:26:40.901749+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:40.901749+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35156492,"num_objects":4292,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":34348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4237","reported_seq":4311,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.889911+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:42.889911+0000","last_peered":"2024-04-26T02:26:42.889911+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:42.889911+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34681356,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4235,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"32'4182","reported_seq":4243,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:40.006638+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:40.006638+0000","last_peered":"2024-04-26T02:26:40.006638+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:40.006638+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34214412,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":33428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":545406120,"num_objects":66584,"num_object_clones":0,"num_object_copies":266336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66611,"num_write_kb":532770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":966868992,"data_stored":966868992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66638,"ondisk_log_size":66638,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481516,"kb_used_data":265188,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93890324,"statfs":{"total":96636764160,"available":96143691776,"internally_reserved":0,"allocated":271552512,"data_stored":271376549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1,5,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483828,"kb_used_data":268012,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93888012,"statfs":{"total":96636764160,"available":96141324288,"internally_reserved":0,"allocated":274444288,"data_stored":274268325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305844,"kb_used_data":143724,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94065996,"statfs":{"total":96636764160,"available":96323579904,"internally_reserved":0,"allocated":147173376,"data_stored":146997413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476916,"kb_used_data":264812,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93894924,"statfs":{"total":96636764160,"available":96148402176,"internally_reserved":0,"allocated":271167488,"data_stored":270991525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217181903},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271335424,"data_stored":271335424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148004864,"data_stored":148004864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275017728,"data_stored":275017728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272510976,"data_stored":272510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:45.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:45.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "32'4035", 'reported_seq': 4073, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.872317+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:40.872317+0000', 'last_peered': '2024-04-26T02:26:40.872317+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:40.872317+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4035, 'log_dups_size': 0, 'ondisk_log_size': 4035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33054720, 'num_objects': 4035, 'num_object_clones': 0, 'num_object_copies': 16140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4035, 'num_write_kb': 32280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2018}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:45.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'4179", 'reported_seq': 4241, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:39.959406+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:39.959406+0000', 'last_peered': '2024-04-26T02:26:39.959406+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:39.959406+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4179, 'log_dups_size': 0, 'ondisk_log_size': 4179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34198028, 'num_objects': 4175, 'num_object_clones': 0, 'num_object_copies': 16700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4177, 'num_write_kb': 33412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "32'4121", 'reported_seq': 4195, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:42.882162+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:42.882162+0000', 'last_peered': '2024-04-26T02:26:42.882162+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:42.882162+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4121, 'log_dups_size': 0, 'ondisk_log_size': 4121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33732870, 'num_objects': 4118, 'num_object_clones': 0, 'num_object_copies': 16472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4120, 'num_write_kb': 32950, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2094}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'3990", 'reported_seq': 4049, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:39.994664+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:39.994664+0000', 'last_peered': '2024-04-26T02:26:39.994664+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:39.994664+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 3990, 'log_dups_size': 0, 'ondisk_log_size': 3990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32649740, 'num_objects': 3986, 'num_object_clones': 0, 'num_object_copies': 15944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3988, 'num_write_kb': 31893, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "32'4030", 'reported_seq': 4082, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.802036+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:40.802036+0000', 'last_peered': '2024-04-26T02:26:40.802036+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:40.802036+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4030, 'log_dups_size': 0, 'ondisk_log_size': 4030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32985612, 'num_objects': 4027, 'num_object_clones': 0, 'num_object_copies': 16108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4028, 'num_write_kb': 32221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2016}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "32'4172", 'reported_seq': 4212, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.907390+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:40.907390+0000', 'last_peered': '2024-04-26T02:26:40.907390+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:40.907390+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4172, 'log_dups_size': 0, 'ondisk_log_size': 4172, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34177024, 'num_objects': 4172, 'num_object_clones': 0, 'num_object_copies': 16688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4172, 'num_write_kb': 33376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2121}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-04-26T02:26:45.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "32'4194", 'reported_seq': 8148, 'reported_epoch': 32, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.029556+0000', 'last_change': '2024-04-26T02:24:59.527447+0000', 'last_active': '2024-04-26T02:26:40.029556+0000', 'last_peered': '2024-04-26T02:26:40.029556+0000', 'last_clean': '2024-04-26T02:23:51.843125+0000', 'last_became_active': '2024-04-26T02:24:59.513563+0000', 'last_became_peered': '2024-04-26T02:24:59.513563+0000', 'last_unstale': '2024-04-26T02:26:40.029556+0000', 'last_undegraded': '2024-04-26T02:24:59.264738+0000', 'last_fullsized': '2024-04-26T02:24:59.234625+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4194, 'log_dups_size': 0, 'ondisk_log_size': 4194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34304524, 'num_objects': 4188, 'num_object_clones': 0, 'num_object_copies': 16752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 139, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4192, 'num_write_kb': 33516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1947, 'num_bytes_recovered': 15917056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4049}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'4323", 'reported_seq': 4384, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.043664+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:40.043664+0000', 'last_peered': '2024-04-26T02:26:40.043664+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:40.043664+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4323, 'log_dups_size': 0, 'ondisk_log_size': 4323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35369484, 'num_objects': 4318, 'num_object_clones': 0, 'num_object_copies': 17272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4321, 'num_write_kb': 34556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "32'4168", 'reported_seq': 4221, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.923911+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:40.923911+0000', 'last_peered': '2024-04-26T02:26:40.923911+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:40.923911+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4168, 'log_dups_size': 0, 'ondisk_log_size': 4168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34071576, 'num_objects': 4160, 'num_object_clones': 0, 'num_object_copies': 16640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4165, 'num_write_kb': 33297, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2156}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "32'4199", 'reported_seq': 4274, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:42.859992+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:42.859992+0000', 'last_peered': '2024-04-26T02:26:42.859992+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:42.859992+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4199, 'log_dups_size': 0, 'ondisk_log_size': 4199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34378252, 'num_objects': 4197, 'num_object_clones': 0, 'num_object_copies': 16788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4197, 'num_write_kb': 33574, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "32'4173", 'reported_seq': 4227, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.931063+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:40.931063+0000', 'last_peered': '2024-04-26T02:26:40.931063+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:40.931063+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4173, 'log_dups_size': 0, 'ondisk_log_size': 4173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34138898, 'num_objects': 4168, 'num_object_clones': 0, 'num_object_copies': 16672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4170, 'num_write_kb': 33348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2151}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:45.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "32'4124", 'reported_seq': 4159, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.954214+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:40.954214+0000', 'last_peered': '2024-04-26T02:26:40.954214+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:40.954214+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4124, 'log_dups_size': 0, 'ondisk_log_size': 4124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33773830, 'num_objects': 4123, 'num_object_clones': 0, 'num_object_copies': 16492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4123, 'num_write_kb': 32983, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "32'4215", 'reported_seq': 4286, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:42.782850+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:42.782850+0000', 'last_peered': '2024-04-26T02:26:42.782850+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:42.782850+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4215, 'log_dups_size': 0, 'ondisk_log_size': 4215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34519302, 'num_objects': 4214, 'num_object_clones': 0, 'num_object_copies': 16856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4214, 'num_write_kb': 33711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "32'4296", 'reported_seq': 4345, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.901749+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:40.901749+0000', 'last_peered': '2024-04-26T02:26:40.901749+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:40.901749+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4296, 'log_dups_size': 0, 'ondisk_log_size': 4296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35156492, 'num_objects': 4292, 'num_object_clones': 0, 'num_object_copies': 17168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4294, 'num_write_kb': 34348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "32'4237", 'reported_seq': 4311, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:42.889911+0000', 'last_change': '2024-04-26T02:24:59.522853+0000', 'last_active': '2024-04-26T02:26:42.889911+0000', 'last_peered': '2024-04-26T02:26:42.889911+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:42.889911+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4237, 'log_dups_size': 0, 'ondisk_log_size': 4237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34681356, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 16936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4235, 'num_write_kb': 33877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:45.649 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "32'4182", 'reported_seq': 4243, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:40.006638+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:40.006638+0000', 'last_peered': '2024-04-26T02:26:40.006638+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:40.006638+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4182, 'log_dups_size': 0, 'ondisk_log_size': 4182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34214412, 'num_objects': 4177, 'num_object_clones': 0, 'num_object_copies': 16708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4180, 'num_write_kb': 33428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:45.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:45.735 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 144 20 67913 67893 3.68195 1.77344 0.00750816 0.0422828 2024-04-26T02:26:46.736 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 145 20 68189 68169 3.6714 2.15625 0.0741711 0.0425458 2024-04-26T02:26:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:46.933 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30658/270820 objects degraded (11.320%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67705,"data_bytes":554589352,"bytes_used":1811140608,"bytes_avail":384735916032,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":30658,"degraded_total":270820,"degraded_ratio":0.11320434236762425,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":305079,"recovering_keys_per_sec":0,"num_objects_recovered":298,"num_bytes_recovered":2441216,"num_keys_recovered":0,"write_bytes_sec":2665858,"read_op_per_sec":0,"write_op_per_sec":325},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:47.022 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:47.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:47.737 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 146 20 68469 68449 3.66123 2.1875 0.12594 0.0426605 2024-04-26T02:26:48.312 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:48.312 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:48.465 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-04-26T02:26:46.474641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554589352,"num_objects":67705,"num_object_clones":0,"num_object_copies":270820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67732,"num_write_kb":541738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67759,"ondisk_log_size":67759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1768692,"kb_used_data":954000,"kb_used_omap":14,"kb_used_meta":814641,"kb_avail":375718668,"statfs":{"total":386547056640,"available":384735916032,"internally_reserved":0,"allocated":976896000,"data_stored":976187579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":834192947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,16,7,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21331968,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":2441216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001913"},"pg_stats":[{"pgid":"1.f","version":"34'4131","reported_seq":4173,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.777940+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:45.777940+0000","last_peered":"2024-04-26T02:26:45.777940+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:45.777940+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33841152,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":33048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'4270","reported_seq":4334,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:44.970558+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:44.970558+0000","last_peered":"2024-04-26T02:26:44.970558+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:44.970558+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34943500,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":34140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"32'4121","reported_seq":4195,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.882162+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:42.882162+0000","last_peered":"2024-04-26T02:26:42.882162+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:42.882162+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33732870,"num_objects":4118,"num_object_clones":0,"num_object_copies":16472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4120,"num_write_kb":32950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'4097","reported_seq":4158,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:44.938917+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:44.938917+0000","last_peered":"2024-04-26T02:26:44.938917+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:44.938917+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33526284,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4120","reported_seq":4176,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.992655+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:45.992655+0000","last_peered":"2024-04-26T02:26:45.992655+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:45.992655+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33722892,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":32941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4259","reported_seq":4303,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.765396+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:45.765396+0000","last_peered":"2024-04-26T02:26:45.765396+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:45.765396+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34889728,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":34072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"32'4276","reported_seq":8508,"reported_epoch":32,"state":"peering","last_fresh":"2024-04-26T02:26:44.701429+0000","last_change":"2024-04-26T02:26:44.765262+0000","last_active":"2024-04-26T02:26:44.765262+0000","last_peered":"2024-04-26T02:26:44.701429+0000","last_clean":"2024-04-26T02:23:51.843125+0000","last_became_active":"2024-04-26T02:24:59.513563+0000","last_became_peered":"2024-04-26T02:24:59.513563+0000","last_unstale":"2024-04-26T02:26:44.701429+0000","last_undegraded":"2024-04-26T02:26:44.701429+0000","last_fullsized":"2024-04-26T02:24:59.234625+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34976268,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4274,"num_write_kb":34172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'4413","reported_seq":4477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:44.852452+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:44.852452+0000","last_peered":"2024-04-26T02:26:44.852452+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:44.852452+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36106764,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":35276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4250","reported_seq":4307,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.887830+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:45.887830+0000","last_peered":"2024-04-26T02:26:45.887830+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:45.887830+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34743320,"num_objects":4242,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":33953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'4199","reported_seq":4274,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.859992+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:42.859992+0000","last_peered":"2024-04-26T02:26:42.859992+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:42.859992+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34378252,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":33574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4262","reported_seq":4318,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.923803+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:45.923803+0000","last_peered":"2024-04-26T02:26:45.923803+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:45.923803+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34867986,"num_objects":4257,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":34060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4229","reported_seq":4268,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.977703+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:45.977703+0000","last_peered":"2024-04-26T02:26:45.977703+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:45.977703+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34633990,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":33823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2240}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'4215","reported_seq":4286,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.782850+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:42.782850+0000","last_peered":"2024-04-26T02:26:42.782850+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:42.782850+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34519302,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":33711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4395","reported_seq":4448,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.783564+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:45.783564+0000","last_peered":"2024-04-26T02:26:45.783564+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:45.783564+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35967500,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":35140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"32'4237","reported_seq":4311,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:42.889911+0000","last_change":"2024-04-26T02:24:59.522853+0000","last_active":"2024-04-26T02:26:42.889911+0000","last_peered":"2024-04-26T02:26:42.889911+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:42.889911+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34681356,"num_objects":4234,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4235,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'4285","reported_seq":4348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:45.010662+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:45.010662+0000","last_peered":"2024-04-26T02:26:45.010662+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:45.010662+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058188,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":34252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":554589352,"num_objects":67705,"num_object_clones":0,"num_object_copies":270820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67732,"num_write_kb":541738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979030016,"data_stored":979030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67759,"ondisk_log_size":67759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491900,"kb_used_data":271408,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879940,"statfs":{"total":96636764160,"available":96133058560,"internally_reserved":0,"allocated":277921792,"data_stored":277741260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483828,"kb_used_data":268012,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93888012,"statfs":{"total":96636764160,"available":96141324288,"internally_reserved":0,"allocated":274444288,"data_stored":274268325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305844,"kb_used_data":143724,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94065996,"statfs":{"total":96636764160,"available":96323579904,"internally_reserved":0,"allocated":147173376,"data_stored":146997413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487120,"kb_used_data":270856,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93884720,"statfs":{"total":96636764160,"available":96137953280,"internally_reserved":0,"allocated":277356544,"data_stored":277180581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221441743},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277544960,"data_stored":277544960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148004864,"data_stored":148004864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275017728,"data_stored":275017728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278462464,"data_stored":278462464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:48.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 147 20 68769 68749 3.65226 2.34375 0.0542857 0.0427675 2024-04-26T02:26:49.737 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 148 20 69080 69060 3.644 2.42969 0.0517137 0.0428621 2024-04-26T02:26:50.737 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 149 20 69375 69355 3.63502 2.30469 0.0652919 0.0429709 2024-04-26T02:26:51.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:51.737 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 150 20 69680 69660 3.62667 2.38281 0.0984536 0.0430702 2024-04-26T02:26:52.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 151 20 69978 69958 3.61807 2.32812 0.0653834 0.0431724 2024-04-26T02:26:52.999 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:52.999 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:53.082 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-04-26T02:26:52.484924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566639784,"num_objects":69176,"num_object_clones":0,"num_object_copies":276704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69203,"num_write_kb":553506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":17530880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69230,"ondisk_log_size":69230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1798128,"kb_used_data":983436,"kb_used_omap":14,"kb_used_meta":814641,"kb_avail":375689232,"statfs":{"total":386547056640,"available":384705773568,"internally_reserved":0,"allocated":1007038464,"data_stored":1006297976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":834192947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,17,13,5,16,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21233664,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":20736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.010725"},"pg_stats":[{"pgid":"1.f","version":"34'4229","reported_seq":4271,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.086309+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:51.086309+0000","last_peered":"2024-04-26T02:26:51.086309+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:51.086309+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34643968,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":33832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4372","reported_seq":4438,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.046579+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:50.046579+0000","last_peered":"2024-04-26T02:26:50.046579+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:50.046579+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35779084,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":34956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4208","reported_seq":4286,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:47.995026+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:47.995026+0000","last_peered":"2024-04-26T02:26:47.995026+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:47.995026+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34445574,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":33646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4202","reported_seq":4265,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.052937+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:50.052937+0000","last_peered":"2024-04-26T02:26:50.052937+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:50.052937+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34386444,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4200,"num_write_kb":33589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4210","reported_seq":4266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.062221+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:51.062221+0000","last_peered":"2024-04-26T02:26:51.062221+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:51.062221+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34460172,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4208,"num_write_kb":33661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4352","reported_seq":4396,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.029244+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:51.029244+0000","last_peered":"2024-04-26T02:26:51.029244+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:51.029244+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":4352,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4352,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4373","reported_seq":8615,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:49.958540+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:49.958540+0000","last_peered":"2024-04-26T02:26:49.958540+0000","last_clean":"2024-04-26T02:26:49.958540+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:49.958540+0000","last_undegraded":"2024-04-26T02:26:49.958540+0000","last_fullsized":"2024-04-26T02:26:49.958540+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:51.595632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35770892,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":34948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4508","reported_seq":4574,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:49.962604+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:49.962604+0000","last_peered":"2024-04-26T02:26:49.962604+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:49.962604+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885004,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":36036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4342","reported_seq":4399,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.075283+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:51.075283+0000","last_peered":"2024-04-26T02:26:51.075283+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:51.075283+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35496984,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":34689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4288","reported_seq":4367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:48.016235+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:48.016235+0000","last_peered":"2024-04-26T02:26:48.016235+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:48.016235+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35107340,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":34286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4333","reported_seq":4391,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.152857+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:51.152857+0000","last_peered":"2024-04-26T02:26:51.152857+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:51.152857+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35449618,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":34628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4321","reported_seq":4360,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.136358+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:51.136358+0000","last_peered":"2024-04-26T02:26:51.136358+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:51.136358+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35387654,"num_objects":4320,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":34559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4303","reported_seq":4378,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:47.946817+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:47.946817+0000","last_peered":"2024-04-26T02:26:47.946817+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:47.946817+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35240198,"num_objects":4302,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":34415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4485","reported_seq":4538,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.015257+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:51.015257+0000","last_peered":"2024-04-26T02:26:51.015257+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:51.015257+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36704780,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4321","reported_seq":4517,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:48.013928+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:26:48.013928+0000","last_peered":"2024-04-26T02:26:48.013928+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:48.013928+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35369484,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":34549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":475136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4383","reported_seq":4448,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.042379+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:50.042379+0000","last_peered":"2024-04-26T02:26:50.042379+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:50.042379+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861004,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566639784,"num_objects":69176,"num_object_clones":0,"num_object_copies":276704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69203,"num_write_kb":553506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":17530880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009569792,"data_stored":1009569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69230,"ondisk_log_size":69230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489260,"kb_used_data":277092,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93882580,"statfs":{"total":96636764160,"available":96135761920,"internally_reserved":0,"allocated":283742208,"data_stored":283557086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,3,1,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493688,"kb_used_data":273708,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93878152,"statfs":{"total":96636764160,"available":96131227648,"internally_reserved":0,"allocated":280276992,"data_stored":280091870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314204,"kb_used_data":156244,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94057636,"statfs":{"total":96636764160,"available":96315019264,"internally_reserved":0,"allocated":159993856,"data_stored":159808734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500976,"kb_used_data":276392,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93870864,"statfs":{"total":96636764160,"available":96123764736,"internally_reserved":0,"allocated":283025408,"data_stored":282840286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283361280,"data_stored":283361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160686080,"data_stored":160686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280850432,"data_stored":280850432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284672000,"data_stored":284672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:53.083 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:53.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 152 20 70269 70249 3.60923 2.27344 0.0621853 0.0432778 2024-04-26T02:26:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:54.489 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:54.570 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-04-26T02:26:52.484924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566639784,"num_objects":69176,"num_object_clones":0,"num_object_copies":276704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69203,"num_write_kb":553506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":17530880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69230,"ondisk_log_size":69230,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1798128,"kb_used_data":983436,"kb_used_omap":14,"kb_used_meta":814641,"kb_avail":375689232,"statfs":{"total":386547056640,"available":384705773568,"internally_reserved":0,"allocated":1007038464,"data_stored":1006297976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":834192947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,17,13,5,16,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21233664,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":20736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.010725"},"pg_stats":[{"pgid":"1.f","version":"34'4229","reported_seq":4271,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.086309+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:51.086309+0000","last_peered":"2024-04-26T02:26:51.086309+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:51.086309+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34643968,"num_objects":4229,"num_object_clones":0,"num_object_copies":16916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":33832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4372","reported_seq":4438,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.046579+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:50.046579+0000","last_peered":"2024-04-26T02:26:50.046579+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:50.046579+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35779084,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":34956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4208","reported_seq":4286,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:47.995026+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:47.995026+0000","last_peered":"2024-04-26T02:26:47.995026+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:47.995026+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34445574,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":33646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2181}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4202","reported_seq":4265,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.052937+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:50.052937+0000","last_peered":"2024-04-26T02:26:50.052937+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:50.052937+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34386444,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4200,"num_write_kb":33589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4210","reported_seq":4266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.062221+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:51.062221+0000","last_peered":"2024-04-26T02:26:51.062221+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:51.062221+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34460172,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4208,"num_write_kb":33661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4352","reported_seq":4396,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.029244+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:51.029244+0000","last_peered":"2024-04-26T02:26:51.029244+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:51.029244+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":4352,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4352,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4373","reported_seq":8615,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:49.958540+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:49.958540+0000","last_peered":"2024-04-26T02:26:49.958540+0000","last_clean":"2024-04-26T02:26:49.958540+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:49.958540+0000","last_undegraded":"2024-04-26T02:26:49.958540+0000","last_fullsized":"2024-04-26T02:26:49.958540+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:51.595632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35770892,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":34948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4508","reported_seq":4574,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:49.962604+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:49.962604+0000","last_peered":"2024-04-26T02:26:49.962604+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:49.962604+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885004,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":36036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4342","reported_seq":4399,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.075283+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:51.075283+0000","last_peered":"2024-04-26T02:26:51.075283+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:51.075283+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35496984,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":34689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4288","reported_seq":4367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:48.016235+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:48.016235+0000","last_peered":"2024-04-26T02:26:48.016235+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:48.016235+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35107340,"num_objects":4286,"num_object_clones":0,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":34286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4333","reported_seq":4391,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.152857+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:51.152857+0000","last_peered":"2024-04-26T02:26:51.152857+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:51.152857+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35449618,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":34628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4321","reported_seq":4360,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.136358+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:51.136358+0000","last_peered":"2024-04-26T02:26:51.136358+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:51.136358+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35387654,"num_objects":4320,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":34559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4303","reported_seq":4378,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:47.946817+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:47.946817+0000","last_peered":"2024-04-26T02:26:47.946817+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:47.946817+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35240198,"num_objects":4302,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":34415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2191}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4485","reported_seq":4538,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:51.015257+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:51.015257+0000","last_peered":"2024-04-26T02:26:51.015257+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:51.015257+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36704780,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4321","reported_seq":4517,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:48.013928+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:26:48.013928+0000","last_peered":"2024-04-26T02:26:48.013928+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:48.013928+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35369484,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":34549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":475136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4383","reported_seq":4448,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:50.042379+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:50.042379+0000","last_peered":"2024-04-26T02:26:50.042379+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:50.042379+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861004,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566639784,"num_objects":69176,"num_object_clones":0,"num_object_copies":276704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69203,"num_write_kb":553506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":17530880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009569792,"data_stored":1009569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69230,"ondisk_log_size":69230,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489260,"kb_used_data":277092,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93882580,"statfs":{"total":96636764160,"available":96135761920,"internally_reserved":0,"allocated":283742208,"data_stored":283557086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,3,1,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493688,"kb_used_data":273708,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93878152,"statfs":{"total":96636764160,"available":96131227648,"internally_reserved":0,"allocated":280276992,"data_stored":280091870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314204,"kb_used_data":156244,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94057636,"statfs":{"total":96636764160,"available":96315019264,"internally_reserved":0,"allocated":159993856,"data_stored":159808734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500976,"kb_used_data":276392,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93870864,"statfs":{"total":96636764160,"available":96123764736,"internally_reserved":0,"allocated":283025408,"data_stored":282840286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283361280,"data_stored":283361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160686080,"data_stored":160686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280850432,"data_stored":280850432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284672000,"data_stored":284672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:54.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'4229", 'reported_seq': 4271, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.086309+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:26:51.086309+0000', 'last_peered': '2024-04-26T02:26:51.086309+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:26:51.086309+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4229, 'log_dups_size': 0, 'ondisk_log_size': 4229, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34643968, 'num_objects': 4229, 'num_object_clones': 0, 'num_object_copies': 16916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4229, 'num_write_kb': 33832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2212}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'4372", 'reported_seq': 4438, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:50.046579+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:26:50.046579+0000', 'last_peered': '2024-04-26T02:26:50.046579+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:26:50.046579+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4372, 'log_dups_size': 0, 'ondisk_log_size': 4372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35779084, 'num_objects': 4368, 'num_object_clones': 0, 'num_object_copies': 17472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4370, 'num_write_kb': 34956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'4208", 'reported_seq': 4286, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:47.995026+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:47.995026+0000', 'last_peered': '2024-04-26T02:26:47.995026+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:47.995026+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4208, 'log_dups_size': 0, 'ondisk_log_size': 4208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34445574, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4207, 'num_write_kb': 33646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2181}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:26:54.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4202", 'reported_seq': 4265, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:50.052937+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:26:50.052937+0000', 'last_peered': '2024-04-26T02:26:50.052937+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:26:50.052937+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4202, 'log_dups_size': 0, 'ondisk_log_size': 4202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34386444, 'num_objects': 4198, 'num_object_clones': 0, 'num_object_copies': 16792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4200, 'num_write_kb': 33589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'4210", 'reported_seq': 4266, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.062221+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:26:51.062221+0000', 'last_peered': '2024-04-26T02:26:51.062221+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:26:51.062221+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4210, 'log_dups_size': 0, 'ondisk_log_size': 4210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34460172, 'num_objects': 4207, 'num_object_clones': 0, 'num_object_copies': 16828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4208, 'num_write_kb': 33661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'4352", 'reported_seq': 4396, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.029244+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:26:51.029244+0000', 'last_peered': '2024-04-26T02:26:51.029244+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:26:51.029244+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4352, 'log_dups_size': 0, 'ondisk_log_size': 4352, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35651584, 'num_objects': 4352, 'num_object_clones': 0, 'num_object_copies': 17408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4352, 'num_write_kb': 34816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2301}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'4508", 'reported_seq': 4574, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:49.962604+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:49.962604+0000', 'last_peered': '2024-04-26T02:26:49.962604+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:49.962604+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4508, 'log_dups_size': 0, 'ondisk_log_size': 4508, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36885004, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 18012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4506, 'num_write_kb': 36036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'4342", 'reported_seq': 4399, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.075283+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:26:51.075283+0000', 'last_peered': '2024-04-26T02:26:51.075283+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:26:51.075283+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4342, 'log_dups_size': 0, 'ondisk_log_size': 4342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35496984, 'num_objects': 4334, 'num_object_clones': 0, 'num_object_copies': 17336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4339, 'num_write_kb': 34689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'4288", 'reported_seq': 4367, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:48.016235+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:48.016235+0000', 'last_peered': '2024-04-26T02:26:48.016235+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:48.016235+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35107340, 'num_objects': 4286, 'num_object_clones': 0, 'num_object_copies': 17144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4286, 'num_write_kb': 34286, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2249}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:54.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'4333", 'reported_seq': 4391, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.152857+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:26:51.152857+0000', 'last_peered': '2024-04-26T02:26:51.152857+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:26:51.152857+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4333, 'log_dups_size': 0, 'ondisk_log_size': 4333, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35449618, 'num_objects': 4328, 'num_object_clones': 0, 'num_object_copies': 17312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4330, 'num_write_kb': 34628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2311}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'4321", 'reported_seq': 4360, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.136358+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:26:51.136358+0000', 'last_peered': '2024-04-26T02:26:51.136358+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:26:51.136358+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4321, 'log_dups_size': 0, 'ondisk_log_size': 4321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35387654, 'num_objects': 4320, 'num_object_clones': 0, 'num_object_copies': 17280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4320, 'num_write_kb': 34559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'4303", 'reported_seq': 4378, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:47.946817+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:47.946817+0000', 'last_peered': '2024-04-26T02:26:47.946817+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:47.946817+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4303, 'log_dups_size': 0, 'ondisk_log_size': 4303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35240198, 'num_objects': 4302, 'num_object_clones': 0, 'num_object_copies': 17208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4302, 'num_write_kb': 34415, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'4485", 'reported_seq': 4538, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:51.015257+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:26:51.015257+0000', 'last_peered': '2024-04-26T02:26:51.015257+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:26:51.015257+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4485, 'log_dups_size': 0, 'ondisk_log_size': 4485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36704780, 'num_objects': 4481, 'num_object_clones': 0, 'num_object_copies': 17924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4483, 'num_write_kb': 35860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'4321", 'reported_seq': 4517, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:48.013928+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:26:48.013928+0000', 'last_peered': '2024-04-26T02:26:48.013928+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:48.013928+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4321, 'log_dups_size': 0, 'ondisk_log_size': 4321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35369484, 'num_objects': 4318, 'num_object_clones': 0, 'num_object_copies': 17272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1995, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4319, 'num_write_kb': 34549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 475136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:26:54.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:26:54.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'4383", 'reported_seq': 4448, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:50.042379+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:26:50.042379+0000', 'last_peered': '2024-04-26T02:26:50.042379+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:26:50.042379+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4383, 'log_dups_size': 0, 'ondisk_log_size': 4383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35861004, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4381, 'num_write_kb': 35036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:26:54.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:26:54.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 153 20 70566 70546 3.60081 2.32031 0.105827 0.0433766 2024-04-26T02:26:55.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 154 20 70852 70832 3.59193 2.23438 0.0522346 0.043483 2024-04-26T02:26:56.313 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:56.314 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30542/278272 objects degraded (10.976%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":212,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69568,"data_bytes":569851048,"bytes_used":1857437696,"bytes_avail":384689618944,"bytes_total":386547056640,"degraded_objects":30542,"degraded_total":278272,"degraded_ratio":0.10975592226310947,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":212929,"recovering_keys_per_sec":0,"num_objects_recovered":253,"num_bytes_recovered":2072576,"num_keys_recovered":0,"write_bytes_sec":2511391,"read_op_per_sec":0,"write_op_per_sec":306},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:26:56.387 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:26:56.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:56.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 155 20 71110 71090 3.58177 2.01562 0.0905586 0.0436099 2024-04-26T02:26:57.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:26:57.252 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:26:57.342 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-26T02:26:56.488647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579214504,"num_objects":70711,"num_object_clones":0,"num_object_copies":282844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70738,"num_write_kb":565786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":17989632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70765,"ondisk_log_size":70765,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1854796,"kb_used_data":1002604,"kb_used_omap":14,"kb_used_meta":852145,"kb_avail":375632564,"statfs":{"total":386547056640,"available":384647745536,"internally_reserved":0,"allocated":1026666496,"data_stored":1025926008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":872597043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,11,7,14,16,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29933568,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.735286"},"pg_stats":[{"pgid":"1.f","version":"34'4329","reported_seq":4370,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.114616+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:56.114616+0000","last_peered":"2024-04-26T02:26:56.114616+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:56.114616+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4463","reported_seq":4528,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:55.051960+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:26:55.051960+0000","last_peered":"2024-04-26T02:26:55.051960+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:26:55.051960+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36524556,"num_objects":4459,"num_object_clones":0,"num_object_copies":17836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4308","reported_seq":4386,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:53.046631+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:53.046631+0000","last_peered":"2024-04-26T02:26:53.046631+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:53.046631+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35264774,"num_objects":4305,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4307,"num_write_kb":34446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4295","reported_seq":4358,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:54.813993+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:26:54.813993+0000","last_peered":"2024-04-26T02:26:54.813993+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:26:54.813993+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35148300,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4293,"num_write_kb":34333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4315","reported_seq":4371,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.052352+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:56.052352+0000","last_peered":"2024-04-26T02:26:56.052352+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:56.052352+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35320332,"num_objects":4312,"num_object_clones":0,"num_object_copies":17248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":34501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4435","reported_seq":4479,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.187231+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:56.187231+0000","last_peered":"2024-04-26T02:26:56.187231+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:56.187231+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36331520,"num_objects":4435,"num_object_clones":0,"num_object_copies":17740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4467","reported_seq":8709,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:54.999980+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:54.999980+0000","last_peered":"2024-04-26T02:26:54.999980+0000","last_clean":"2024-04-26T02:26:54.999980+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:54.999980+0000","last_undegraded":"2024-04-26T02:26:54.999980+0000","last_fullsized":"2024-04-26T02:26:54.999980+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:57.599231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36540940,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4592","reported_seq":4658,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:55.011023+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:55.011023+0000","last_peered":"2024-04-26T02:26:55.011023+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:55.011023+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573132,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4439","reported_seq":4496,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.197771+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:56.197771+0000","last_peered":"2024-04-26T02:26:56.197771+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:56.197771+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36291608,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4383","reported_seq":4462,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:52.961757+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:52.961757+0000","last_peered":"2024-04-26T02:26:52.961757+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:52.961757+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35885580,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2344}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4430","reported_seq":4486,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.188121+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:56.188121+0000","last_peered":"2024-04-26T02:26:56.188121+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:56.188121+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36244242,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":35404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4408","reported_seq":4447,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.172015+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:56.172015+0000","last_peered":"2024-04-26T02:26:56.172015+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:56.172015+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36100358,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4387","reported_seq":4462,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:53.095668+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:53.095668+0000","last_peered":"2024-04-26T02:26:53.095668+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:53.095668+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35928326,"num_objects":4386,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4386,"num_write_kb":35087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4593","reported_seq":4646,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.200669+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:56.200669+0000","last_peered":"2024-04-26T02:26:56.200669+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:56.200669+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37589516,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4434","reported_seq":4742,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:53.061047+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:26:53.061047+0000","last_peered":"2024-04-26T02:26:53.061047+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:53.061047+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295180,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4487","reported_seq":4552,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:54.983710+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:26:54.983710+0000","last_peered":"2024-04-26T02:26:54.983710+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:26:54.983710+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36712972,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4485,"num_write_kb":35868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":579214504,"num_objects":70711,"num_object_clones":0,"num_object_copies":282844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70738,"num_write_kb":565786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":17989632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028030464,"data_stored":1028030464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70765,"ondisk_log_size":70765,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518528,"kb_used_data":284152,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93853312,"statfs":{"total":96636764160,"available":96105791488,"internally_reserved":0,"allocated":290971648,"data_stored":290786526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":239988432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,5,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509464,"kb_used_data":279760,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93862376,"statfs":{"total":96636764160,"available":96115073024,"internally_reserved":0,"allocated":286474240,"data_stored":286289118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314204,"kb_used_data":156244,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94057636,"statfs":{"total":96636764160,"available":96315019264,"internally_reserved":0,"allocated":159993856,"data_stored":159808734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512600,"kb_used_data":282448,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93859240,"statfs":{"total":96636764160,"available":96111861760,"internally_reserved":0,"allocated":289226752,"data_stored":289041630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289452032,"data_stored":289452032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160686080,"data_stored":160686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287096832,"data_stored":287096832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290795520,"data_stored":290795520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:57.738 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 156 20 71357 71337 3.57118 1.92969 0.0322918 0.0437143 2024-04-26T02:26:58.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 157 20 71659 71639 3.56342 2.35938 0.0407697 0.0438332 2024-04-26T02:26:59.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 158 20 71952 71932 3.55536 2.28906 0.11412 0.0439247 2024-04-26T02:27:00.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:00.741 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 159 20 72199 72179 3.54514 1.92969 0.0542625 0.0440504 2024-04-26T02:27:01.627 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:01.627 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:01.692 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-26T02:27:00.495796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585219240,"num_objects":71444,"num_object_clones":0,"num_object_copies":285776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71471,"num_write_kb":571650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2297,"num_bytes_recovered":18579456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71498,"ondisk_log_size":71498,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1920708,"kb_used_data":1019940,"kb_used_omap":14,"kb_used_meta":900721,"kb_avail":375566652,"statfs":{"total":386547056640,"available":384580251648,"internally_reserved":0,"allocated":1044418560,"data_stored":1043678072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":922338867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,8,11,7,8,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27779072,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":27128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.020622"},"pg_stats":[{"pgid":"1.f","version":"34'4329","reported_seq":4370,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.114616+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:26:56.114616+0000","last_peered":"2024-04-26T02:26:56.114616+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:26:56.114616+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35463168,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":34632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4554","reported_seq":4619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.116918+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:00.116918+0000","last_peered":"2024-04-26T02:27:00.116918+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:00.116918+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270028,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4552,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4378","reported_seq":4455,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.088162+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:58.088162+0000","last_peered":"2024-04-26T02:26:58.088162+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:58.088162+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35838214,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":35006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2351}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4378","reported_seq":4439,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.052637+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:00.052637+0000","last_peered":"2024-04-26T02:27:00.052637+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:00.052637+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35828236,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":34997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4315","reported_seq":4371,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.052352+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:26:56.052352+0000","last_peered":"2024-04-26T02:26:56.052352+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:26:56.052352+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35320332,"num_objects":4312,"num_object_clones":0,"num_object_copies":17248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":34501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4435","reported_seq":4479,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.187231+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:26:56.187231+0000","last_peered":"2024-04-26T02:26:56.187231+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:26:56.187231+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36331520,"num_objects":4435,"num_object_clones":0,"num_object_copies":17740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":35480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4538","reported_seq":8780,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:59.956006+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:59.956006+0000","last_peered":"2024-04-26T02:26:59.956006+0000","last_clean":"2024-04-26T02:26:59.956006+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:59.956006+0000","last_undegraded":"2024-04-26T02:26:59.956006+0000","last_fullsized":"2024-04-26T02:26:59.956006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:02.733937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37122572,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":36268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4679","reported_seq":4745,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:59.944452+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:59.944452+0000","last_peered":"2024-04-26T02:26:59.944452+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:59.944452+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38285836,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4439","reported_seq":4496,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.197771+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:26:56.197771+0000","last_peered":"2024-04-26T02:26:56.197771+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:26:56.197771+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36291608,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4466","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.111163+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:58.111163+0000","last_peered":"2024-04-26T02:26:58.111163+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:58.111163+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36565516,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4430","reported_seq":4486,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.188121+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:26:56.188121+0000","last_peered":"2024-04-26T02:26:56.188121+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:26:56.188121+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36244242,"num_objects":4425,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":35404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4408","reported_seq":4447,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.172015+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:26:56.172015+0000","last_peered":"2024-04-26T02:26:56.172015+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:26:56.172015+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36100358,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4473","reported_seq":4548,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.071165+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:58.071165+0000","last_peered":"2024-04-26T02:26:58.071165+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:58.071165+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36632838,"num_objects":4472,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4472,"num_write_kb":35775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4593","reported_seq":4646,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:56.200669+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:26:56.200669+0000","last_peered":"2024-04-26T02:26:56.200669+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:26:56.200669+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37589516,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4509","reported_seq":4961,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.154095+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:26:58.154095+0000","last_peered":"2024-04-26T02:26:58.154095+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:58.154095+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36909580,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4507,"num_write_kb":36053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4574","reported_seq":4639,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.141742+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:00.141742+0000","last_peered":"2024-04-26T02:27:00.141742+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:00.141742+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37425676,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4572,"num_write_kb":36564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":585219240,"num_objects":71444,"num_object_clones":0,"num_object_copies":285776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71471,"num_write_kb":571650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2297,"num_bytes_recovered":18579456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1045475328,"data_stored":1045475328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71498,"ondisk_log_size":71498,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518528,"kb_used_data":284152,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93853312,"statfs":{"total":96636764160,"available":96105791488,"internally_reserved":0,"allocated":290971648,"data_stored":290786526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":239988432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,5,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537380,"kb_used_data":285468,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93834460,"statfs":{"total":96636764160,"available":96086487040,"internally_reserved":0,"allocated":292319232,"data_stored":292134110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324384,"kb_used_data":162264,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94047456,"statfs":{"total":96636764160,"available":96304594944,"internally_reserved":0,"allocated":166158336,"data_stored":165973214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540416,"kb_used_data":288056,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93831424,"statfs":{"total":96636764160,"available":96083378176,"internally_reserved":0,"allocated":294969344,"data_stored":294784222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295043072,"data_stored":295043072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166817792,"data_stored":166817792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292818944,"data_stored":292818944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290795520,"data_stored":290795520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:01.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:01.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:27:01.741061+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0441605 2024-04-26T02:27:01.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:27:01.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 160 20 72488 72468 3.53709 2.25781 0.0712062 0.0441605 2024-04-26T02:27:02.740 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 161 20 72772 72752 3.5289 2.21875 0.0564461 0.0442617 2024-04-26T02:27:03.223 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:03.224 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:03.270 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-04-26T02:27:02.496381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590101672,"num_objects":72040,"num_object_clones":0,"num_object_copies":288160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72067,"num_write_kb":576418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2297,"num_bytes_recovered":18579456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72094,"ondisk_log_size":72094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1962472,"kb_used_data":1031176,"kb_used_omap":14,"kb_used_meta":931249,"kb_avail":375524888,"statfs":{"total":386547056640,"available":384537485312,"internally_reserved":0,"allocated":1055924224,"data_stored":1055183736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":953599539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,7,11,6,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28590080,"num_objects":3490,"num_object_clones":0,"num_object_copies":13960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":27920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.020665"},"pg_stats":[{"pgid":"1.f","version":"34'4404","reported_seq":4446,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.189610+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:01.189610+0000","last_peered":"2024-04-26T02:27:01.189610+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:01.189610+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36077568,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":35232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4554","reported_seq":4619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.116918+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:00.116918+0000","last_peered":"2024-04-26T02:27:00.116918+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:00.116918+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270028,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4552,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4378","reported_seq":4455,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.088162+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:26:58.088162+0000","last_peered":"2024-04-26T02:26:58.088162+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:26:58.088162+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35838214,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":35006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2351}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4378","reported_seq":4439,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.052637+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:00.052637+0000","last_peered":"2024-04-26T02:27:00.052637+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:00.052637+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35828236,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":34997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4406","reported_seq":4462,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.200248+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:01.200248+0000","last_peered":"2024-04-26T02:27:01.200248+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:01.200248+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36065804,"num_objects":4403,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":35229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4526","reported_seq":4570,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.003910+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:01.003910+0000","last_peered":"2024-04-26T02:27:01.003910+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:01.003910+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37076992,"num_objects":4526,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4538","reported_seq":8780,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:59.956006+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:59.956006+0000","last_peered":"2024-04-26T02:26:59.956006+0000","last_clean":"2024-04-26T02:26:59.956006+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:59.956006+0000","last_undegraded":"2024-04-26T02:26:59.956006+0000","last_fullsized":"2024-04-26T02:26:59.956006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:02.733937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37122572,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":36268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4679","reported_seq":4745,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:59.944452+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:59.944452+0000","last_peered":"2024-04-26T02:26:59.944452+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:59.944452+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38285836,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4513","reported_seq":4570,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.217458+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:01.217458+0000","last_peered":"2024-04-26T02:27:01.217458+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:01.217458+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36897816,"num_objects":4505,"num_object_clones":0,"num_object_copies":18020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4466","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.111163+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:26:58.111163+0000","last_peered":"2024-04-26T02:26:58.111163+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:26:58.111163+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36565516,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4526","reported_seq":4584,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.101210+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:01.101210+0000","last_peered":"2024-04-26T02:27:01.101210+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:01.101210+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37030674,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4523,"num_write_kb":36172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4494","reported_seq":4533,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.133387+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:01.133387+0000","last_peered":"2024-04-26T02:27:01.133387+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:01.133387+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36804870,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":35943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4473","reported_seq":4548,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.071165+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:26:58.071165+0000","last_peered":"2024-04-26T02:26:58.071165+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:26:58.071165+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36632838,"num_objects":4472,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4472,"num_write_kb":35775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4676","reported_seq":4729,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.245619+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:01.245619+0000","last_peered":"2024-04-26T02:27:01.245619+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:01.245619+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38269452,"num_objects":4672,"num_object_clones":0,"num_object_copies":18688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4674,"num_write_kb":37388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4509","reported_seq":4961,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:58.154095+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:26:58.154095+0000","last_peered":"2024-04-26T02:26:58.154095+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:26:58.154095+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36909580,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4507,"num_write_kb":36053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2639}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4574","reported_seq":4639,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.141742+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:00.141742+0000","last_peered":"2024-04-26T02:27:00.141742+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:00.141742+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37425676,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4572,"num_write_kb":36564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590101672,"num_objects":72040,"num_object_clones":0,"num_object_copies":288160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72067,"num_write_kb":576418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2297,"num_bytes_recovered":18579456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056927744,"data_stored":1056927744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72094,"ondisk_log_size":72094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550304,"kb_used_data":289560,"kb_used_omap":4,"kb_used_meta":260731,"kb_avail":93821536,"statfs":{"total":96636764160,"available":96073252864,"internally_reserved":0,"allocated":296509440,"data_stored":296324318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":266989264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537380,"kb_used_data":285468,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93834460,"statfs":{"total":96636764160,"available":96086487040,"internally_reserved":0,"allocated":292319232,"data_stored":292134110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334372,"kb_used_data":168092,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94037468,"statfs":{"total":96636764160,"available":96294367232,"internally_reserved":0,"allocated":172126208,"data_stored":171941086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540416,"kb_used_data":288056,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93831424,"statfs":{"total":96636764160,"available":96083378176,"internally_reserved":0,"allocated":294969344,"data_stored":294784222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295043072,"data_stored":295043072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172703744,"data_stored":172703744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292818944,"data_stored":292818944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296361984,"data_stored":296361984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'4404", 'reported_seq': 4446, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.189610+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:01.189610+0000', 'last_peered': '2024-04-26T02:27:01.189610+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:01.189610+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4404, 'log_dups_size': 0, 'ondisk_log_size': 4404, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36077568, 'num_objects': 4404, 'num_object_clones': 0, 'num_object_copies': 17616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4404, 'num_write_kb': 35232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2387}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'4554", 'reported_seq': 4619, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:00.116918+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:00.116918+0000', 'last_peered': '2024-04-26T02:27:00.116918+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:00.116918+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4554, 'log_dups_size': 0, 'ondisk_log_size': 4554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37270028, 'num_objects': 4550, 'num_object_clones': 0, 'num_object_copies': 18200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4552, 'num_write_kb': 36412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'4378", 'reported_seq': 4455, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:58.088162+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:26:58.088162+0000', 'last_peered': '2024-04-26T02:26:58.088162+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:26:58.088162+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4378, 'log_dups_size': 0, 'ondisk_log_size': 4378, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35838214, 'num_objects': 4375, 'num_object_clones': 0, 'num_object_copies': 17500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4377, 'num_write_kb': 35006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2351}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:03.271 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4378", 'reported_seq': 4439, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:00.052637+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:00.052637+0000', 'last_peered': '2024-04-26T02:27:00.052637+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:00.052637+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4378, 'log_dups_size': 0, 'ondisk_log_size': 4378, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35828236, 'num_objects': 4374, 'num_object_clones': 0, 'num_object_copies': 17496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4376, 'num_write_kb': 34997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'4406", 'reported_seq': 4462, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.200248+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:01.200248+0000', 'last_peered': '2024-04-26T02:27:01.200248+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:01.200248+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4406, 'log_dups_size': 0, 'ondisk_log_size': 4406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36065804, 'num_objects': 4403, 'num_object_clones': 0, 'num_object_copies': 17612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4404, 'num_write_kb': 35229, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'4526", 'reported_seq': 4570, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.003910+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:01.003910+0000', 'last_peered': '2024-04-26T02:27:01.003910+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:01.003910+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4526, 'log_dups_size': 0, 'ondisk_log_size': 4526, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37076992, 'num_objects': 4526, 'num_object_clones': 0, 'num_object_copies': 18104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4526, 'num_write_kb': 36208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2475}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'4679", 'reported_seq': 4745, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:59.944452+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:26:59.944452+0000', 'last_peered': '2024-04-26T02:26:59.944452+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:26:59.944452+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4679, 'log_dups_size': 0, 'ondisk_log_size': 4679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38285836, 'num_objects': 4674, 'num_object_clones': 0, 'num_object_copies': 18696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4677, 'num_write_kb': 37404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2533}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'4513", 'reported_seq': 4570, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.217458+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:01.217458+0000', 'last_peered': '2024-04-26T02:27:01.217458+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:01.217458+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4513, 'log_dups_size': 0, 'ondisk_log_size': 4513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36897816, 'num_objects': 4505, 'num_object_clones': 0, 'num_object_copies': 18020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4510, 'num_write_kb': 36057, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2501}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:03.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'4466", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:58.111163+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:26:58.111163+0000', 'last_peered': '2024-04-26T02:26:58.111163+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:26:58.111163+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4466, 'log_dups_size': 0, 'ondisk_log_size': 4466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36565516, 'num_objects': 4464, 'num_object_clones': 0, 'num_object_copies': 17856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4464, 'num_write_kb': 35710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'4526", 'reported_seq': 4584, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.101210+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:01.101210+0000', 'last_peered': '2024-04-26T02:27:01.101210+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:01.101210+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4526, 'log_dups_size': 0, 'ondisk_log_size': 4526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37030674, 'num_objects': 4521, 'num_object_clones': 0, 'num_object_copies': 18084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4523, 'num_write_kb': 36172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2504}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'4494", 'reported_seq': 4533, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.133387+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:01.133387+0000', 'last_peered': '2024-04-26T02:27:01.133387+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:01.133387+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4494, 'log_dups_size': 0, 'ondisk_log_size': 4494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36804870, 'num_objects': 4493, 'num_object_clones': 0, 'num_object_copies': 17972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4493, 'num_write_kb': 35943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'4473", 'reported_seq': 4548, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:58.071165+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:26:58.071165+0000', 'last_peered': '2024-04-26T02:26:58.071165+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:26:58.071165+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4473, 'log_dups_size': 0, 'ondisk_log_size': 4473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36632838, 'num_objects': 4472, 'num_object_clones': 0, 'num_object_copies': 17888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4472, 'num_write_kb': 35775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'4676", 'reported_seq': 4729, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:01.245619+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:01.245619+0000', 'last_peered': '2024-04-26T02:27:01.245619+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:01.245619+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4676, 'log_dups_size': 0, 'ondisk_log_size': 4676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38269452, 'num_objects': 4672, 'num_object_clones': 0, 'num_object_copies': 18688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4674, 'num_write_kb': 37388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'4509", 'reported_seq': 4961, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:26:58.154095+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:26:58.154095+0000', 'last_peered': '2024-04-26T02:26:58.154095+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:26:58.154095+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4509, 'log_dups_size': 0, 'ondisk_log_size': 4509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36909580, 'num_objects': 4506, 'num_object_clones': 0, 'num_object_copies': 18024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1867, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4507, 'num_write_kb': 36053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 188, 'num_bytes_recovered': 1523712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:03.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:03.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'4574", 'reported_seq': 4639, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:00.141742+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:00.141742+0000', 'last_peered': '2024-04-26T02:27:00.141742+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:00.141742+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4574, 'log_dups_size': 0, 'ondisk_log_size': 4574, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37425676, 'num_objects': 4569, 'num_object_clones': 0, 'num_object_copies': 18276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4572, 'num_write_kb': 36564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:03.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:03.742 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 162 20 73056 73036 3.52079 2.21875 0.0979359 0.0443649 2024-04-26T02:27:04.742 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 163 20 73332 73312 3.51242 2.15625 0.0644738 0.0444679 2024-04-26T02:27:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:04.967 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30416/289600 objects degraded (10.503%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":221,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72400,"data_bytes":593050792,"bytes_used":2019622912,"bytes_avail":384527433728,"bytes_total":386547056640,"degraded_objects":30416,"degraded_total":289600,"degraded_ratio":0.10502762430939226,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":124121,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"write_bytes_sec":2198719,"read_op_per_sec":0,"write_op_per_sec":268},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:27:05.120 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:05.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:05.742 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 164 20 73616 73596 3.50453 2.21875 0.036644 0.0445679 2024-04-26T02:27:06.351 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:06.352 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:06.428 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-26T02:27:04.496917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593050792,"num_objects":72400,"num_object_clones":0,"num_object_copies":289600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72427,"num_write_kb":579298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":19021824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72454,"ondisk_log_size":72454,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1972288,"kb_used_data":1036832,"kb_used_omap":14,"kb_used_meta":935409,"kb_avail":375515072,"statfs":{"total":386547056640,"available":384527433728,"internally_reserved":0,"allocated":1061715968,"data_stored":1060975480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":957859379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,7,8,10,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26411008,"num_objects":3224,"num_object_clones":0,"num_object_copies":12896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011992"},"pg_stats":[{"pgid":"1.f","version":"34'4404","reported_seq":4446,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.189610+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:01.189610+0000","last_peered":"2024-04-26T02:27:01.189610+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:01.189610+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36077568,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":35232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4554","reported_seq":4619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.116918+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:00.116918+0000","last_peered":"2024-04-26T02:27:00.116918+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:00.116918+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37270028,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4552,"num_write_kb":36412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4483","reported_seq":4560,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:03.189162+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:03.189162+0000","last_peered":"2024-04-26T02:27:03.189162+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:03.189162+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36698374,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":35846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2456}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4378","reported_seq":4439,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.052637+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:00.052637+0000","last_peered":"2024-04-26T02:27:00.052637+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:00.052637+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35828236,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":34997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4406","reported_seq":4462,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.200248+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:01.200248+0000","last_peered":"2024-04-26T02:27:01.200248+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:01.200248+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36065804,"num_objects":4403,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":35229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4526","reported_seq":4570,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.003910+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:01.003910+0000","last_peered":"2024-04-26T02:27:01.003910+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:01.003910+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37076992,"num_objects":4526,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4538","reported_seq":8780,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:26:59.956006+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:26:59.956006+0000","last_peered":"2024-04-26T02:26:59.956006+0000","last_clean":"2024-04-26T02:26:59.956006+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:26:59.956006+0000","last_undegraded":"2024-04-26T02:26:59.956006+0000","last_fullsized":"2024-04-26T02:26:59.956006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:02.733937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37122572,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":36268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4679","reported_seq":4745,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:26:59.944452+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:26:59.944452+0000","last_peered":"2024-04-26T02:26:59.944452+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:26:59.944452+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38285836,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4513","reported_seq":4570,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.217458+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:01.217458+0000","last_peered":"2024-04-26T02:27:01.217458+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:01.217458+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36897816,"num_objects":4505,"num_object_clones":0,"num_object_copies":18020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4550","reported_seq":4627,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:03.146260+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:03.146260+0000","last_peered":"2024-04-26T02:27:03.146260+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:03.146260+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253644,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":36382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4526","reported_seq":4584,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.101210+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:01.101210+0000","last_peered":"2024-04-26T02:27:01.101210+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:01.101210+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37030674,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4523,"num_write_kb":36172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2504}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4494","reported_seq":4533,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.133387+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:01.133387+0000","last_peered":"2024-04-26T02:27:01.133387+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:01.133387+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36804870,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":35943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4552","reported_seq":4627,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:03.072503+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:03.072503+0000","last_peered":"2024-04-26T02:27:03.072503+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:03.072503+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37280006,"num_objects":4551,"num_object_clones":0,"num_object_copies":18204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":36407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4676","reported_seq":4729,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:01.245619+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:01.245619+0000","last_peered":"2024-04-26T02:27:01.245619+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:01.245619+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38269452,"num_objects":4672,"num_object_clones":0,"num_object_copies":18688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4674,"num_write_kb":37388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4601","reported_seq":5161,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:03.173230+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:03.173230+0000","last_peered":"2024-04-26T02:27:03.173230+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:03.173230+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4601,"log_dups_size":0,"ondisk_log_size":4601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37663244,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":36789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2785}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4574","reported_seq":4639,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:00.141742+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:00.141742+0000","last_peered":"2024-04-26T02:27:00.141742+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:00.141742+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37425676,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4572,"num_write_kb":36564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":593050792,"num_objects":72400,"num_object_clones":0,"num_object_copies":289600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72427,"num_write_kb":579298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":19021824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1062703104,"data_stored":1062703104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72454,"ondisk_log_size":72454,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550304,"kb_used_data":289560,"kb_used_omap":4,"kb_used_meta":260731,"kb_avail":93821536,"statfs":{"total":96636764160,"available":96073252864,"internally_reserved":0,"allocated":296509440,"data_stored":296324318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":266989264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547196,"kb_used_data":291124,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93824644,"statfs":{"total":96636764160,"available":96076435456,"internally_reserved":0,"allocated":298110976,"data_stored":297925854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334372,"kb_used_data":168092,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94037468,"statfs":{"total":96636764160,"available":96294367232,"internally_reserved":0,"allocated":172126208,"data_stored":171941086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540416,"kb_used_data":288056,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93831424,"statfs":{"total":96636764160,"available":96083378176,"internally_reserved":0,"allocated":294969344,"data_stored":294784222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295043072,"data_stored":295043072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172703744,"data_stored":172703744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298594304,"data_stored":298594304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296361984,"data_stored":296361984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:06.742 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 165 20 73903 73883 3.49688 2.24219 0.137824 0.0446662 2024-04-26T02:27:07.743 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 166 20 74197 74177 3.48966 2.29688 0.0857672 0.0447567 2024-04-26T02:27:08.742 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 167 20 74486 74466 3.48228 2.25781 0.034096 0.0448542 2024-04-26T02:27:09.430 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:09.745 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 168 20 74771 74751 3.47474 2.22656 0.0540544 0.0449506 2024-04-26T02:27:10.588 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:10.589 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:10.711 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-04-26T02:27:10.499012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608156840,"num_objects":74244,"num_object_clones":0,"num_object_copies":296976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74271,"num_write_kb":594050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2412,"num_bytes_recovered":19521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74298,"ondisk_log_size":74298,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1959136,"kb_used_data":1065280,"kb_used_omap":14,"kb_used_meta":893809,"kb_avail":375528224,"statfs":{"total":386547056640,"available":384540901376,"internally_reserved":0,"allocated":1090846720,"data_stored":1090106232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":915260979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,13,14,11,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26370048,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005471"},"pg_stats":[{"pgid":"1.f","version":"34'4493","reported_seq":4535,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.208658+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:06.208658+0000","last_peered":"2024-04-26T02:27:06.208658+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:06.208658+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36806656,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":35944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4714","reported_seq":4780,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.070452+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:10.070452+0000","last_peered":"2024-04-26T02:27:10.070452+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:10.070452+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38580748,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4571","reported_seq":4649,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.119212+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:08.119212+0000","last_peered":"2024-04-26T02:27:08.119212+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:08.119212+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37419270,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":36550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4542","reported_seq":4605,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:09.881660+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:09.881660+0000","last_peered":"2024-04-26T02:27:09.881660+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:09.881660+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37171724,"num_objects":4538,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":36309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4499","reported_seq":4555,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.275831+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:06.275831+0000","last_peered":"2024-04-26T02:27:06.275831+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:06.275831+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36827660,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4602","reported_seq":4646,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.185927+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:06.185927+0000","last_peered":"2024-04-26T02:27:06.185927+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:06.185927+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37699584,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":36816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4709","reported_seq":8951,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:10.143699+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:10.143699+0000","last_peered":"2024-04-26T02:27:10.143699+0000","last_clean":"2024-04-26T02:27:10.143699+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:10.143699+0000","last_undegraded":"2024-04-26T02:27:10.143699+0000","last_fullsized":"2024-04-26T02:27:10.143699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:14.690576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38523404,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4865","reported_seq":4931,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.049398+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:10.049398+0000","last_peered":"2024-04-26T02:27:10.049398+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:10.049398+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39809548,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4604","reported_seq":4661,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.090650+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:06.090650+0000","last_peered":"2024-04-26T02:27:06.090650+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:06.090650+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37643288,"num_objects":4596,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":36785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4629","reported_seq":4708,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:07.997408+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:07.997408+0000","last_peered":"2024-04-26T02:27:07.997408+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:07.997408+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900812,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4627,"num_write_kb":37014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4620","reported_seq":4678,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.177619+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:06.177619+0000","last_peered":"2024-04-26T02:27:06.177619+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:06.177619+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37800722,"num_objects":4615,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4599","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.311611+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:06.311611+0000","last_peered":"2024-04-26T02:27:06.311611+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:06.311611+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37665030,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4598,"num_write_kb":36783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4642","reported_seq":4717,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.172749+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:08.172749+0000","last_peered":"2024-04-26T02:27:08.172749+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:08.172749+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38017286,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":37127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4776","reported_seq":4829,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.254295+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:06.254295+0000","last_peered":"2024-04-26T02:27:06.254295+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:06.254295+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39088652,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4774,"num_write_kb":38188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2701}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4695","reported_seq":5376,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.258391+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:08.258391+0000","last_peered":"2024-04-26T02:27:08.258391+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:08.258391+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433292,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4693,"num_write_kb":37541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":2465792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4738","reported_seq":4803,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.133903+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:10.133903+0000","last_peered":"2024-04-26T02:27:10.133903+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:10.133903+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769164,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2669}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":608156840,"num_objects":74244,"num_object_clones":0,"num_object_copies":296976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74271,"num_write_kb":594050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2412,"num_bytes_recovered":19521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1092268032,"data_stored":1092268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74298,"ondisk_log_size":74298,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543588,"kb_used_data":295324,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93828252,"statfs":{"total":96636764160,"available":96080130048,"internally_reserved":0,"allocated":302411776,"data_stored":302226654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,6,4,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540324,"kb_used_data":296732,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93831516,"statfs":{"total":96636764160,"available":96083472384,"internally_reserved":0,"allocated":303853568,"data_stored":303668446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327740,"kb_used_data":173940,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94044100,"statfs":{"total":96636764160,"available":96301158400,"internally_reserved":0,"allocated":178114560,"data_stored":177929438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547484,"kb_used_data":299284,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93824356,"statfs":{"total":96636764160,"available":96076140544,"internally_reserved":0,"allocated":306466816,"data_stored":306281694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306782208,"data_stored":306782208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178769920,"data_stored":178769920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304517120,"data_stored":304517120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":302198784,"data_stored":302198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:10.712 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:10.745 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 169 20 75039 75019 3.46657 2.09375 0.0384735 0.0450563 2024-04-26T02:27:11.746 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 170 20 75329 75309 3.45951 2.26562 0.0487795 0.0451483 2024-04-26T02:27:12.329 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:12.329 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:12.399 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-04-26T02:27:10.499012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608156840,"num_objects":74244,"num_object_clones":0,"num_object_copies":296976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74271,"num_write_kb":594050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2412,"num_bytes_recovered":19521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74298,"ondisk_log_size":74298,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1959136,"kb_used_data":1065280,"kb_used_omap":14,"kb_used_meta":893809,"kb_avail":375528224,"statfs":{"total":386547056640,"available":384540901376,"internally_reserved":0,"allocated":1090846720,"data_stored":1090106232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":915260979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,13,14,11,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26370048,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005471"},"pg_stats":[{"pgid":"1.f","version":"34'4493","reported_seq":4535,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.208658+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:06.208658+0000","last_peered":"2024-04-26T02:27:06.208658+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:06.208658+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36806656,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":35944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4714","reported_seq":4780,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.070452+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:10.070452+0000","last_peered":"2024-04-26T02:27:10.070452+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:10.070452+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38580748,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4571","reported_seq":4649,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.119212+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:08.119212+0000","last_peered":"2024-04-26T02:27:08.119212+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:08.119212+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37419270,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":36550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4542","reported_seq":4605,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:09.881660+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:09.881660+0000","last_peered":"2024-04-26T02:27:09.881660+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:09.881660+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37171724,"num_objects":4538,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":36309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4499","reported_seq":4555,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.275831+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:06.275831+0000","last_peered":"2024-04-26T02:27:06.275831+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:06.275831+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36827660,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4602","reported_seq":4646,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.185927+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:06.185927+0000","last_peered":"2024-04-26T02:27:06.185927+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:06.185927+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37699584,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":36816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4709","reported_seq":8951,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:10.143699+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:10.143699+0000","last_peered":"2024-04-26T02:27:10.143699+0000","last_clean":"2024-04-26T02:27:10.143699+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:10.143699+0000","last_undegraded":"2024-04-26T02:27:10.143699+0000","last_fullsized":"2024-04-26T02:27:10.143699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:14.690576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38523404,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4865","reported_seq":4931,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.049398+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:10.049398+0000","last_peered":"2024-04-26T02:27:10.049398+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:10.049398+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39809548,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4604","reported_seq":4661,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.090650+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:06.090650+0000","last_peered":"2024-04-26T02:27:06.090650+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:06.090650+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37643288,"num_objects":4596,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":36785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4629","reported_seq":4708,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:07.997408+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:07.997408+0000","last_peered":"2024-04-26T02:27:07.997408+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:07.997408+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900812,"num_objects":4627,"num_object_clones":0,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4627,"num_write_kb":37014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4620","reported_seq":4678,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.177619+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:06.177619+0000","last_peered":"2024-04-26T02:27:06.177619+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:06.177619+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37800722,"num_objects":4615,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4599","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.311611+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:06.311611+0000","last_peered":"2024-04-26T02:27:06.311611+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:06.311611+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37665030,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4598,"num_write_kb":36783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4642","reported_seq":4717,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.172749+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:08.172749+0000","last_peered":"2024-04-26T02:27:08.172749+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:08.172749+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38017286,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":37127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4776","reported_seq":4829,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:06.254295+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:06.254295+0000","last_peered":"2024-04-26T02:27:06.254295+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:06.254295+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39088652,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4774,"num_write_kb":38188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2701}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4695","reported_seq":5376,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:08.258391+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:08.258391+0000","last_peered":"2024-04-26T02:27:08.258391+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:08.258391+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38433292,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4693,"num_write_kb":37541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":2465792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4738","reported_seq":4803,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.133903+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:10.133903+0000","last_peered":"2024-04-26T02:27:10.133903+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:10.133903+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769164,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2669}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":608156840,"num_objects":74244,"num_object_clones":0,"num_object_copies":296976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74271,"num_write_kb":594050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2412,"num_bytes_recovered":19521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1092268032,"data_stored":1092268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74298,"ondisk_log_size":74298,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543588,"kb_used_data":295324,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93828252,"statfs":{"total":96636764160,"available":96080130048,"internally_reserved":0,"allocated":302411776,"data_stored":302226654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,6,4,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540324,"kb_used_data":296732,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93831516,"statfs":{"total":96636764160,"available":96083472384,"internally_reserved":0,"allocated":303853568,"data_stored":303668446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327740,"kb_used_data":173940,"kb_used_omap":4,"kb_used_meta":153787,"kb_avail":94044100,"statfs":{"total":96636764160,"available":96301158400,"internally_reserved":0,"allocated":178114560,"data_stored":177929438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157478607},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547484,"kb_used_data":299284,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93824356,"statfs":{"total":96636764160,"available":96076140544,"internally_reserved":0,"allocated":306466816,"data_stored":306281694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306782208,"data_stored":306782208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178769920,"data_stored":178769920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304517120,"data_stored":304517120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":302198784,"data_stored":302198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'4493", 'reported_seq': 4535, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.208658+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:06.208658+0000', 'last_peered': '2024-04-26T02:27:06.208658+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:06.208658+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4493, 'log_dups_size': 0, 'ondisk_log_size': 4493, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36806656, 'num_objects': 4493, 'num_object_clones': 0, 'num_object_copies': 17972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4493, 'num_write_kb': 35944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2476}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'4714", 'reported_seq': 4780, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:10.070452+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:10.070452+0000', 'last_peered': '2024-04-26T02:27:10.070452+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:10.070452+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4714, 'log_dups_size': 0, 'ondisk_log_size': 4714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38580748, 'num_objects': 4710, 'num_object_clones': 0, 'num_object_copies': 18840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4712, 'num_write_kb': 37692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'4571", 'reported_seq': 4649, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:08.119212+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:08.119212+0000', 'last_peered': '2024-04-26T02:27:08.119212+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:08.119212+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4571, 'log_dups_size': 0, 'ondisk_log_size': 4571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37419270, 'num_objects': 4568, 'num_object_clones': 0, 'num_object_copies': 18272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4570, 'num_write_kb': 36550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:12.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4542", 'reported_seq': 4605, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:09.881660+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:09.881660+0000', 'last_peered': '2024-04-26T02:27:09.881660+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:09.881660+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4542, 'log_dups_size': 0, 'ondisk_log_size': 4542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37171724, 'num_objects': 4538, 'num_object_clones': 0, 'num_object_copies': 18152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4540, 'num_write_kb': 36309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'4499", 'reported_seq': 4555, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.275831+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:06.275831+0000', 'last_peered': '2024-04-26T02:27:06.275831+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:06.275831+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4499, 'log_dups_size': 0, 'ondisk_log_size': 4499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36827660, 'num_objects': 4496, 'num_object_clones': 0, 'num_object_copies': 17984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4497, 'num_write_kb': 35973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2485}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'4602", 'reported_seq': 4646, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.185927+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:06.185927+0000', 'last_peered': '2024-04-26T02:27:06.185927+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:06.185927+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4602, 'log_dups_size': 0, 'ondisk_log_size': 4602, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37699584, 'num_objects': 4602, 'num_object_clones': 0, 'num_object_copies': 18408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4602, 'num_write_kb': 36816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2551}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'4865", 'reported_seq': 4931, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:10.049398+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:10.049398+0000', 'last_peered': '2024-04-26T02:27:10.049398+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:10.049398+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4865, 'log_dups_size': 0, 'ondisk_log_size': 4865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39809548, 'num_objects': 4860, 'num_object_clones': 0, 'num_object_copies': 19440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4863, 'num_write_kb': 38892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'4604", 'reported_seq': 4661, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.090650+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:06.090650+0000', 'last_peered': '2024-04-26T02:27:06.090650+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:06.090650+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4604, 'log_dups_size': 0, 'ondisk_log_size': 4604, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37643288, 'num_objects': 4596, 'num_object_clones': 0, 'num_object_copies': 18384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4601, 'num_write_kb': 36785, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:12.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'4629", 'reported_seq': 4708, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:07.997408+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:07.997408+0000', 'last_peered': '2024-04-26T02:27:07.997408+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:07.997408+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4629, 'log_dups_size': 0, 'ondisk_log_size': 4629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37900812, 'num_objects': 4627, 'num_object_clones': 0, 'num_object_copies': 18508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4627, 'num_write_kb': 37014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2590}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'4620", 'reported_seq': 4678, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.177619+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:06.177619+0000', 'last_peered': '2024-04-26T02:27:06.177619+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:06.177619+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4620, 'log_dups_size': 0, 'ondisk_log_size': 4620, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37800722, 'num_objects': 4615, 'num_object_clones': 0, 'num_object_copies': 18460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4617, 'num_write_kb': 36924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'4599", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.311611+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:06.311611+0000', 'last_peered': '2024-04-26T02:27:06.311611+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:06.311611+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4599, 'log_dups_size': 0, 'ondisk_log_size': 4599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37665030, 'num_objects': 4598, 'num_object_clones': 0, 'num_object_copies': 18392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4598, 'num_write_kb': 36783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2610}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'4642", 'reported_seq': 4717, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:08.172749+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:08.172749+0000', 'last_peered': '2024-04-26T02:27:08.172749+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:08.172749+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4642, 'log_dups_size': 0, 'ondisk_log_size': 4642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38017286, 'num_objects': 4641, 'num_object_clones': 0, 'num_object_copies': 18564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4641, 'num_write_kb': 37127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'4776", 'reported_seq': 4829, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:06.254295+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:06.254295+0000', 'last_peered': '2024-04-26T02:27:06.254295+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:06.254295+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4776, 'log_dups_size': 0, 'ondisk_log_size': 4776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39088652, 'num_objects': 4772, 'num_object_clones': 0, 'num_object_copies': 19088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4774, 'num_write_kb': 38188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'4695", 'reported_seq': 5376, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:08.258391+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:08.258391+0000', 'last_peered': '2024-04-26T02:27:08.258391+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:08.258391+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4695, 'log_dups_size': 0, 'ondisk_log_size': 4695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38433292, 'num_objects': 4692, 'num_object_clones': 0, 'num_object_copies': 18768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4693, 'num_write_kb': 37541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 303, 'num_bytes_recovered': 2465792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:12.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:12.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'4738", 'reported_seq': 4803, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:10.133903+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:10.133903+0000', 'last_peered': '2024-04-26T02:27:10.133903+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:10.133903+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4738, 'log_dups_size': 0, 'ondisk_log_size': 4738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38769164, 'num_objects': 4733, 'num_object_clones': 0, 'num_object_copies': 18932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4736, 'num_write_kb': 37876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2669}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:12.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:12.746 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 171 20 75620 75600 3.45257 2.27344 0.0812187 0.0452373 2024-04-26T02:27:13.746 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 172 20 75898 75878 3.44513 2.17188 0.0580649 0.045336 2024-04-26T02:27:14.030 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:14.030 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30355/299740 objects degraded (10.127%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":230,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74935,"data_bytes":613817512,"bytes_used":2049323008,"bytes_avail":384497733632,"bytes_total":386547056640,"degraded_objects":30355,"degraded_total":299740,"degraded_ratio":0.10127110162140522,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":78478,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"write_bytes_sec":2382322,"read_op_per_sec":0,"write_op_per_sec":290},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:27:14.047 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:14.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:14.746 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 173 20 76143 76123 3.43628 1.91406 0.0329375 0.0454523 2024-04-26T02:27:15.476 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:15.477 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:15.569 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-04-26T02:27:14.500694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616570024,"num_objects":75271,"num_object_clones":0,"num_object_copies":301084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75298,"num_write_kb":602266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":19996672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75325,"ondisk_log_size":75325,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2015200,"kb_used_data":1082496,"kb_used_omap":14,"kb_used_meta":932657,"kb_avail":375472160,"statfs":{"total":386547056640,"available":384483491840,"internally_reserved":0,"allocated":1108475904,"data_stored":1107735416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":955041331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,16,13,13,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26468352,"num_objects":3231,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":25848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004313"},"pg_stats":[{"pgid":"1.f","version":"34'4599","reported_seq":4640,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.415965+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:11.415965+0000","last_peered":"2024-04-26T02:27:11.415965+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:11.415965+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37675008,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":36792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4714","reported_seq":4780,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.070452+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:10.070452+0000","last_peered":"2024-04-26T02:27:10.070452+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:10.070452+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38580748,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4661","reported_seq":4739,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:13.063520+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:13.063520+0000","last_peered":"2024-04-26T02:27:13.063520+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:13.063520+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156550,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4660,"num_write_kb":37270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4542","reported_seq":4605,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:09.881660+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:09.881660+0000","last_peered":"2024-04-26T02:27:09.881660+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:09.881660+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37171724,"num_objects":4538,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":36309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4605","reported_seq":4661,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.274654+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:11.274654+0000","last_peered":"2024-04-26T02:27:11.274654+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:11.274654+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37696012,"num_objects":4602,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4603,"num_write_kb":36821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4674","reported_seq":4718,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.385084+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:11.385084+0000","last_peered":"2024-04-26T02:27:11.385084+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:11.385084+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38289408,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4674,"num_write_kb":37392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2623}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4709","reported_seq":8951,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:10.143699+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:10.143699+0000","last_peered":"2024-04-26T02:27:10.143699+0000","last_clean":"2024-04-26T02:27:10.143699+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:10.143699+0000","last_undegraded":"2024-04-26T02:27:10.143699+0000","last_fullsized":"2024-04-26T02:27:10.143699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:14.690576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38523404,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4865","reported_seq":4931,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.049398+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:10.049398+0000","last_peered":"2024-04-26T02:27:10.049398+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:10.049398+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4865,"log_dups_size":0,"ondisk_log_size":4865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39809548,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4697","reported_seq":4754,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.388664+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:11.388664+0000","last_peered":"2024-04-26T02:27:11.388664+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:11.388664+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38405144,"num_objects":4689,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4696","reported_seq":4775,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:13.219130+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:13.219130+0000","last_peered":"2024-04-26T02:27:13.219130+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:13.219130+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38449676,"num_objects":4694,"num_object_clones":0,"num_object_copies":18776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4694,"num_write_kb":37550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4723","reported_seq":4781,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.344895+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:11.344895+0000","last_peered":"2024-04-26T02:27:11.344895+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:11.344895+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38644498,"num_objects":4718,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":37748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4690","reported_seq":4729,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.274396+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:11.274396+0000","last_peered":"2024-04-26T02:27:11.274396+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:11.274396+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38410502,"num_objects":4689,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":37511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2701}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4741","reported_seq":4815,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:13.252849+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:13.252849+0000","last_peered":"2024-04-26T02:27:13.252849+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:13.252849+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38828294,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4740,"num_write_kb":37919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4896","reported_seq":4949,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:11.312664+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:11.312664+0000","last_peered":"2024-04-26T02:27:11.312664+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:11.312664+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40071692,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":39148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4775","reported_seq":5573,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:13.167873+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:13.167873+0000","last_peered":"2024-04-26T02:27:13.167873+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:13.167873+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39088652,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4773,"num_write_kb":38181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2940928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3078}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4738","reported_seq":4803,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:10.133903+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:10.133903+0000","last_peered":"2024-04-26T02:27:10.133903+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:10.133903+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38769164,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2669}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616570024,"num_objects":75271,"num_object_clones":0,"num_object_copies":301084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75298,"num_write_kb":602266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":19996672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110253568,"data_stored":1110253568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75325,"ondisk_log_size":75325,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553504,"kb_used_data":301080,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93818336,"statfs":{"total":96636764160,"available":96069976064,"internally_reserved":0,"allocated":308305920,"data_stored":308120798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,2,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554232,"kb_used_data":302320,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93817608,"statfs":{"total":96636764160,"available":96069230592,"internally_reserved":0,"allocated":309575680,"data_stored":309390558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359980,"kb_used_data":179812,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":94011860,"statfs":{"total":96636764160,"available":96268144640,"internally_reserved":0,"allocated":184127488,"data_stored":183942366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184479439},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547484,"kb_used_data":299284,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93824356,"statfs":{"total":96636764160,"available":96076140544,"internally_reserved":0,"allocated":306466816,"data_stored":306281694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306782208,"data_stored":306782208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184954880,"data_stored":184954880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308187136,"data_stored":308187136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:15.748 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 174 20 76421 76401 3.42898 2.17188 0.0444622 0.0455504 2024-04-26T02:27:16.749 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 175 20 76715 76695 3.42248 2.29688 0.0940182 0.0456398 2024-04-26T02:27:17.749 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 176 20 77007 76987 3.416 2.28125 0.0767035 0.0457226 2024-04-26T02:27:18.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:18.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 177 20 77292 77272 3.40928 2.22656 0.0683158 0.045813 2024-04-26T02:27:19.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 178 20 77574 77554 3.4025 2.20312 0.0683571 0.0459066 2024-04-26T02:27:20.061 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:20.061 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:20.129 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-04-26T02:27:18.501726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":627907752,"num_objects":76655,"num_object_clones":0,"num_object_copies":306620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76682,"num_write_kb":613338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":20602880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76709,"ondisk_log_size":76709,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2077528,"kb_used_data":1105976,"kb_used_omap":14,"kb_used_meta":971505,"kb_avail":375409832,"statfs":{"total":386547056640,"available":384419667968,"internally_reserved":0,"allocated":1132519424,"data_stored":1131778936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":994821683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,12,7,12,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26025984,"num_objects":3177,"num_object_clones":0,"num_object_copies":12708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":25416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004231"},"pg_stats":[{"pgid":"1.f","version":"34'4683","reported_seq":4725,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.358717+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:16.358717+0000","last_peered":"2024-04-26T02:27:16.358717+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:16.358717+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38363136,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4815","reported_seq":4881,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:15.084033+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:15.084033+0000","last_peered":"2024-04-26T02:27:15.084033+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:15.084033+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408140,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4743","reported_seq":4821,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.068754+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:18.068754+0000","last_peered":"2024-04-26T02:27:18.068754+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:18.068754+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38828294,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2716}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4645","reported_seq":4708,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:15.164321+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:15.164321+0000","last_peered":"2024-04-26T02:27:15.164321+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:15.164321+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38015500,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":37133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4701","reported_seq":4757,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.428366+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:16.428366+0000","last_peered":"2024-04-26T02:27:16.428366+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:16.428366+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38482444,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4760","reported_seq":4804,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.435886+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:16.435886+0000","last_peered":"2024-04-26T02:27:16.435886+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:16.435886+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38993920,"num_objects":4760,"num_object_clones":0,"num_object_copies":19040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4760,"num_write_kb":38080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4792","reported_seq":9034,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:14.987813+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:14.987813+0000","last_peered":"2024-04-26T02:27:14.987813+0000","last_clean":"2024-04-26T02:27:14.987813+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:14.987813+0000","last_undegraded":"2024-04-26T02:27:14.987813+0000","last_fullsized":"2024-04-26T02:27:14.987813+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:19.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39203340,"num_objects":4786,"num_object_clones":0,"num_object_copies":19144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":38300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'4951","reported_seq":5017,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:15.104903+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:15.104903+0000","last_peered":"2024-04-26T02:27:15.104903+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:15.104903+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40514060,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4949,"num_write_kb":39580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4786","reported_seq":4843,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.324304+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:16.324304+0000","last_peered":"2024-04-26T02:27:16.324304+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:16.324304+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39134232,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4783,"num_write_kb":38241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4777","reported_seq":4856,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.241248+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:18.241248+0000","last_peered":"2024-04-26T02:27:18.241248+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:18.241248+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113228,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":38198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2738}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4793","reported_seq":4851,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.133590+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:16.133590+0000","last_peered":"2024-04-26T02:27:16.133590+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:16.133590+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39217938,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":38308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4770","reported_seq":4809,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.423243+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:16.423243+0000","last_peered":"2024-04-26T02:27:16.423243+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:16.423243+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39065862,"num_objects":4769,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4769,"num_write_kb":38151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2781}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4827","reported_seq":4900,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.206816+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:18.206816+0000","last_peered":"2024-04-26T02:27:18.206816+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:18.206816+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39532806,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4826,"num_write_kb":38607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4990","reported_seq":5043,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.420015+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:16.420015+0000","last_peered":"2024-04-26T02:27:16.420015+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:16.420015+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40841740,"num_objects":4986,"num_object_clones":0,"num_object_copies":19944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4988,"num_write_kb":39900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4867","reported_seq":5813,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.241883+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:18.241883+0000","last_peered":"2024-04-26T02:27:18.241883+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:18.241883+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39842316,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":3547136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4809","reported_seq":4874,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:15.131142+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:15.131142+0000","last_peered":"2024-04-26T02:27:15.131142+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:15.131142+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39350796,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4807,"num_write_kb":38444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":627907752,"num_objects":76655,"num_object_clones":0,"num_object_copies":306620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76682,"num_write_kb":613338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":20602880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1133289472,"data_stored":1133289472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76709,"ondisk_log_size":76709,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567228,"kb_used_data":306484,"kb_used_omap":4,"kb_used_meta":260731,"kb_avail":93804612,"statfs":{"total":96636764160,"available":96055922688,"internally_reserved":0,"allocated":313839616,"data_stored":313654494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":266989264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,3,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563968,"kb_used_data":307896,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93807872,"statfs":{"total":96636764160,"available":96059260928,"internally_reserved":0,"allocated":315285504,"data_stored":315100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389096,"kb_used_data":186720,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93982744,"statfs":{"total":96636764160,"available":96238329856,"internally_reserved":0,"allocated":191201280,"data_stored":191016158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557236,"kb_used_data":304876,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93814604,"statfs":{"total":96636764160,"available":96066154496,"internally_reserved":0,"allocated":312193024,"data_stored":312007902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,7,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312377344,"data_stored":312377344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190951424,"data_stored":190951424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":316026880,"data_stored":316026880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313933824,"data_stored":313933824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:20.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:20.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 179 20 77848 77828 3.39545 2.14062 0.160507 0.0460003 2024-04-26T02:27:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:21.644 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:21.737 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-04-26T02:27:20.502155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631594152,"num_objects":77105,"num_object_clones":0,"num_object_copies":308420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77132,"num_write_kb":616938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":20602880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77159,"ondisk_log_size":77159,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2074984,"kb_used_data":1111752,"kb_used_omap":14,"kb_used_meta":963185,"kb_avail":375412376,"statfs":{"total":386547056640,"available":384422273024,"internally_reserved":0,"allocated":1138434048,"data_stored":1137693560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":986302003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,9,6,5,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26836992,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"1.f","version":"34'4683","reported_seq":4725,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.358717+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:16.358717+0000","last_peered":"2024-04-26T02:27:16.358717+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:16.358717+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38363136,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4920","reported_seq":4986,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:20.141216+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:20.141216+0000","last_peered":"2024-04-26T02:27:20.141216+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:20.141216+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40268300,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4743","reported_seq":4821,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.068754+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:18.068754+0000","last_peered":"2024-04-26T02:27:18.068754+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:18.068754+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38828294,"num_objects":4740,"num_object_clones":0,"num_object_copies":18960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2716}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4736","reported_seq":4799,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:20.166845+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:20.166845+0000","last_peered":"2024-04-26T02:27:20.166845+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:20.166845+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38760972,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4701","reported_seq":4757,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.428366+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:16.428366+0000","last_peered":"2024-04-26T02:27:16.428366+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:16.428366+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38482444,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4760","reported_seq":4804,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.435886+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:16.435886+0000","last_peered":"2024-04-26T02:27:16.435886+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:16.435886+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38993920,"num_objects":4760,"num_object_clones":0,"num_object_copies":19040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4760,"num_write_kb":38080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4881","reported_seq":9123,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:20.202524+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:20.202524+0000","last_peered":"2024-04-26T02:27:20.202524+0000","last_clean":"2024-04-26T02:27:20.202524+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:20.202524+0000","last_undegraded":"2024-04-26T02:27:20.202524+0000","last_fullsized":"2024-04-26T02:27:20.202524+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:24.834703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39932428,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5027","reported_seq":5093,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:19.975575+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:19.975575+0000","last_peered":"2024-04-26T02:27:19.975575+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:19.975575+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41136652,"num_objects":5022,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":40188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4786","reported_seq":4843,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.324304+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:16.324304+0000","last_peered":"2024-04-26T02:27:16.324304+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:16.324304+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39134232,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4783,"num_write_kb":38241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4777","reported_seq":4856,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.241248+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:18.241248+0000","last_peered":"2024-04-26T02:27:18.241248+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:18.241248+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39113228,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":38198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2738}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4793","reported_seq":4851,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.133590+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:16.133590+0000","last_peered":"2024-04-26T02:27:16.133590+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:16.133590+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39217938,"num_objects":4788,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":38308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4770","reported_seq":4809,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.423243+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:16.423243+0000","last_peered":"2024-04-26T02:27:16.423243+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:16.423243+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39065862,"num_objects":4769,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4769,"num_write_kb":38151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2781}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4827","reported_seq":4900,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.206816+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:18.206816+0000","last_peered":"2024-04-26T02:27:18.206816+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:18.206816+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39532806,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4826,"num_write_kb":38607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'4990","reported_seq":5043,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:16.420015+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:16.420015+0000","last_peered":"2024-04-26T02:27:16.420015+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:16.420015+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40841740,"num_objects":4986,"num_object_clones":0,"num_object_copies":19944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4988,"num_write_kb":39900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4867","reported_seq":5813,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:18.241883+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:18.241883+0000","last_peered":"2024-04-26T02:27:18.241883+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:18.241883+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39842316,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":3547136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4898","reported_seq":4963,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:19.894684+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:19.894684+0000","last_peered":"2024-04-26T02:27:19.894684+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:19.894684+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40079884,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":39156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":631594152,"num_objects":77105,"num_object_clones":0,"num_object_copies":308420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77132,"num_write_kb":616938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":20602880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1139163136,"data_stored":1139163136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77159,"ondisk_log_size":77159,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567228,"kb_used_data":306484,"kb_used_omap":4,"kb_used_meta":260731,"kb_avail":93804612,"statfs":{"total":96636764160,"available":96055922688,"internally_reserved":0,"allocated":313839616,"data_stored":313654494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":266989264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,3,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563968,"kb_used_data":307896,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93807872,"statfs":{"total":96636764160,"available":96059260928,"internally_reserved":0,"allocated":315285504,"data_stored":315100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389096,"kb_used_data":186720,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93982744,"statfs":{"total":96636764160,"available":96238329856,"internally_reserved":0,"allocated":191201280,"data_stored":191016158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554692,"kb_used_data":310652,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93817148,"statfs":{"total":96636764160,"available":96068759552,"internally_reserved":0,"allocated":318107648,"data_stored":317922526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249884367},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318251008,"data_stored":318251008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190951424,"data_stored":190951424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":316026880,"data_stored":316026880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313933824,"data_stored":313933824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:21.738 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:21.738 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'4683", 'reported_seq': 4725, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.358717+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:16.358717+0000', 'last_peered': '2024-04-26T02:27:16.358717+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:16.358717+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4683, 'log_dups_size': 0, 'ondisk_log_size': 4683, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38363136, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4683, 'num_write_kb': 37464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2666}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.738 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:21.738 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'4920", 'reported_seq': 4986, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:20.141216+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:20.141216+0000', 'last_peered': '2024-04-26T02:27:20.141216+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:20.141216+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4920, 'log_dups_size': 0, 'ondisk_log_size': 4920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40268300, 'num_objects': 4916, 'num_object_clones': 0, 'num_object_copies': 19664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4918, 'num_write_kb': 39340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'4743", 'reported_seq': 4821, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:18.068754+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:18.068754+0000', 'last_peered': '2024-04-26T02:27:18.068754+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:18.068754+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4743, 'log_dups_size': 0, 'ondisk_log_size': 4743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38828294, 'num_objects': 4740, 'num_object_clones': 0, 'num_object_copies': 18960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4742, 'num_write_kb': 37926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2716}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4736", 'reported_seq': 4799, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:20.166845+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:20.166845+0000', 'last_peered': '2024-04-26T02:27:20.166845+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:20.166845+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4736, 'log_dups_size': 0, 'ondisk_log_size': 4736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38760972, 'num_objects': 4732, 'num_object_clones': 0, 'num_object_copies': 18928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4734, 'num_write_kb': 37861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'4701", 'reported_seq': 4757, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.428366+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:16.428366+0000', 'last_peered': '2024-04-26T02:27:16.428366+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:16.428366+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4701, 'log_dups_size': 0, 'ondisk_log_size': 4701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38482444, 'num_objects': 4698, 'num_object_clones': 0, 'num_object_copies': 18792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 37589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2687}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'4760", 'reported_seq': 4804, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.435886+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:16.435886+0000', 'last_peered': '2024-04-26T02:27:16.435886+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:16.435886+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4760, 'log_dups_size': 0, 'ondisk_log_size': 4760, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38993920, 'num_objects': 4760, 'num_object_clones': 0, 'num_object_copies': 19040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4760, 'num_write_kb': 38080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2709}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:21.739 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5027", 'reported_seq': 5093, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:19.975575+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:19.975575+0000', 'last_peered': '2024-04-26T02:27:19.975575+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:19.975575+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5027, 'log_dups_size': 0, 'ondisk_log_size': 5027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41136652, 'num_objects': 5022, 'num_object_clones': 0, 'num_object_copies': 20088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5025, 'num_write_kb': 40188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'4786", 'reported_seq': 4843, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.324304+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:16.324304+0000', 'last_peered': '2024-04-26T02:27:16.324304+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:16.324304+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4786, 'log_dups_size': 0, 'ondisk_log_size': 4786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39134232, 'num_objects': 4778, 'num_object_clones': 0, 'num_object_copies': 19112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4783, 'num_write_kb': 38241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'4777", 'reported_seq': 4856, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:18.241248+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:18.241248+0000', 'last_peered': '2024-04-26T02:27:18.241248+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:18.241248+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4777, 'log_dups_size': 0, 'ondisk_log_size': 4777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39113228, 'num_objects': 4775, 'num_object_clones': 0, 'num_object_copies': 19100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4775, 'num_write_kb': 38198, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2738}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'4793", 'reported_seq': 4851, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.133590+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:16.133590+0000', 'last_peered': '2024-04-26T02:27:16.133590+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:16.133590+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4793, 'log_dups_size': 0, 'ondisk_log_size': 4793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39217938, 'num_objects': 4788, 'num_object_clones': 0, 'num_object_copies': 19152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4790, 'num_write_kb': 38308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2771}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'4770", 'reported_seq': 4809, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.423243+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:16.423243+0000', 'last_peered': '2024-04-26T02:27:16.423243+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:16.423243+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4770, 'log_dups_size': 0, 'ondisk_log_size': 4770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39065862, 'num_objects': 4769, 'num_object_clones': 0, 'num_object_copies': 19076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4769, 'num_write_kb': 38151, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2781}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'4827", 'reported_seq': 4900, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:18.206816+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:18.206816+0000', 'last_peered': '2024-04-26T02:27:18.206816+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:18.206816+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4827, 'log_dups_size': 0, 'ondisk_log_size': 4827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39532806, 'num_objects': 4826, 'num_object_clones': 0, 'num_object_copies': 19304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4826, 'num_write_kb': 38607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:21.740 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'4990", 'reported_seq': 5043, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:16.420015+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:16.420015+0000', 'last_peered': '2024-04-26T02:27:16.420015+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:16.420015+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4990, 'log_dups_size': 0, 'ondisk_log_size': 4990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40841740, 'num_objects': 4986, 'num_object_clones': 0, 'num_object_copies': 19944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4988, 'num_write_kb': 39900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:21.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:21.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'4867", 'reported_seq': 5813, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:18.241883+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:18.241883+0000', 'last_peered': '2024-04-26T02:27:18.241883+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:18.241883+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4867, 'log_dups_size': 0, 'ondisk_log_size': 4867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39842316, 'num_objects': 4864, 'num_object_clones': 0, 'num_object_copies': 19456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1620, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4865, 'num_write_kb': 38917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 435, 'num_bytes_recovered': 3547136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:21.741 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:21.741 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'4898", 'reported_seq': 4963, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:19.894684+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:19.894684+0000', 'last_peered': '2024-04-26T02:27:19.894684+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:19.894684+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4898, 'log_dups_size': 0, 'ondisk_log_size': 4898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40079884, 'num_objects': 4893, 'num_object_clones': 0, 'num_object_copies': 19572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4896, 'num_write_kb': 39156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:21.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:21.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:27:21.750591+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.046098 2024-04-26T02:27:21.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:27:21.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 180 20 78119 78099 3.38835 2.11719 0.0352995 0.046098 2024-04-26T02:27:22.750 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 181 20 78398 78378 3.38168 2.17969 0.110954 0.0461884 2024-04-26T02:27:23.640 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:23.640 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30223/311020 objects degraded (9.717%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":240,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77755,"data_bytes":636918952,"bytes_used":2111160320,"bytes_avail":384435896320,"bytes_total":386547056640,"degraded_objects":30223,"degraded_total":311020,"degraded_ratio":0.097173815188733845,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":90084,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"write_bytes_sec":2396109,"read_op_per_sec":0,"write_op_per_sec":292},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-26T02:26:34.471700+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:27:23.751 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 182 20 78679 78659 3.37515 2.19531 0.0191555 0.0462766 2024-04-26T02:27:23.807 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:23.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:24.752 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 183 20 78946 78926 3.36809 2.08594 0.0454085 0.046376 2024-04-26T02:27:24.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:24.971 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:25.097 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-04-26T02:27:24.503703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":639925416,"num_objects":78122,"num_object_clones":0,"num_object_copies":312488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78149,"num_write_kb":625074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2602,"num_bytes_recovered":21078016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78176,"ondisk_log_size":78176,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2059952,"kb_used_data":1130000,"kb_used_omap":14,"kb_used_meta":929905,"kb_avail":375427408,"statfs":{"total":386547056640,"available":384437665792,"internally_reserved":0,"allocated":1157120000,"data_stored":1156379512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":952223283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,14,3,9,3,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26107904,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":25496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"1.f","version":"34'4775","reported_seq":4817,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.375951+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:21.375951+0000","last_peered":"2024-04-26T02:27:21.375951+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:21.375951+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39116800,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":38200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4920","reported_seq":4986,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:20.141216+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:20.141216+0000","last_peered":"2024-04-26T02:27:20.141216+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:20.141216+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40268300,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4834","reported_seq":4911,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:23.309070+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:23.309070+0000","last_peered":"2024-04-26T02:27:23.309070+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:23.309070+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39573766,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2807}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4736","reported_seq":4799,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:20.166845+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:20.166845+0000","last_peered":"2024-04-26T02:27:20.166845+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:20.166845+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38760972,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":37861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4802","reported_seq":4857,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.541439+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:21.541439+0000","last_peered":"2024-04-26T02:27:21.541439+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:21.541439+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39309836,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4800,"num_write_kb":38397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2788}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4856","reported_seq":4900,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.500175+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:21.500175+0000","last_peered":"2024-04-26T02:27:21.500175+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:21.500175+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":38848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4881","reported_seq":9123,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:20.202524+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:20.202524+0000","last_peered":"2024-04-26T02:27:20.202524+0000","last_clean":"2024-04-26T02:27:20.202524+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:20.202524+0000","last_undegraded":"2024-04-26T02:27:20.202524+0000","last_fullsized":"2024-04-26T02:27:20.202524+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:24.834703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39932428,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5027","reported_seq":5093,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:19.975575+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:19.975575+0000","last_peered":"2024-04-26T02:27:19.975575+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:19.975575+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41136652,"num_objects":5022,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5025,"num_write_kb":40188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4874","reported_seq":4931,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.523464+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:21.523464+0000","last_peered":"2024-04-26T02:27:21.523464+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:21.523464+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39855128,"num_objects":4866,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4866","reported_seq":4945,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:23.157976+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:23.157976+0000","last_peered":"2024-04-26T02:27:23.157976+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:23.157976+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39842316,"num_objects":4864,"num_object_clones":0,"num_object_copies":19456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4864,"num_write_kb":38910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4888","reported_seq":4946,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.541342+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:21.541342+0000","last_peered":"2024-04-26T02:27:21.541342+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:21.541342+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39996178,"num_objects":4883,"num_object_clones":0,"num_object_copies":19532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":39068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4851","reported_seq":4890,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.501958+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:21.501958+0000","last_peered":"2024-04-26T02:27:21.501958+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:21.501958+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39729414,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4850,"num_write_kb":38799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'4916","reported_seq":4991,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:23.237424+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:23.237424+0000","last_peered":"2024-04-26T02:27:23.237424+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:23.237424+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40261894,"num_objects":4915,"num_object_clones":0,"num_object_copies":19660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":39319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5087","reported_seq":5139,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:21.552276+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:21.552276+0000","last_peered":"2024-04-26T02:27:21.552276+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:21.552276+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41636364,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3012}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'4965","reported_seq":6027,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:23.212126+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:23.212126+0000","last_peered":"2024-04-26T02:27:23.212126+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:23.212126+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40645132,"num_objects":4962,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4963,"num_write_kb":39701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3400}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4898","reported_seq":4963,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:19.894684+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:19.894684+0000","last_peered":"2024-04-26T02:27:19.894684+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:19.894684+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40079884,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":39156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":639925416,"num_objects":78122,"num_object_clones":0,"num_object_copies":312488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78149,"num_write_kb":625074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2602,"num_bytes_recovered":21078016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1156808704,"data_stored":1156808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78176,"ondisk_log_size":78176,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560484,"kb_used_data":312220,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93811356,"statfs":{"total":96636764160,"available":96062828544,"internally_reserved":0,"allocated":319713280,"data_stored":319528158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,5,0,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562240,"kb_used_data":314488,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93809600,"statfs":{"total":96636764160,"available":96061030400,"internally_reserved":0,"allocated":322035712,"data_stored":321850590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,0,1,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382536,"kb_used_data":192640,"kb_used_omap":4,"kb_used_meta":189883,"kb_avail":93989304,"statfs":{"total":96636764160,"available":96245047296,"internally_reserved":0,"allocated":197263360,"data_stored":197078238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194440911},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554692,"kb_used_data":310652,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93817148,"statfs":{"total":96636764160,"available":96068759552,"internally_reserved":0,"allocated":318107648,"data_stored":317922526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249884367},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":318251008,"data_stored":318251008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196988928,"data_stored":196988928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321765376,"data_stored":321765376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319803392,"data_stored":319803392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:25.754 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 184 20 79214 79194 3.36113 2.09375 0.0260098 0.0464741 2024-04-26T02:27:26.754 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 185 20 79515 79495 3.35567 2.35156 0.0443381 0.0465447 2024-04-26T02:27:27.756 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 186 20 79792 79772 3.34923 2.16406 0.0635126 0.0466388 2024-04-26T02:27:28.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:28.756 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 187 20 80065 80045 3.34272 2.13281 0.0505823 0.0467203 2024-04-26T02:27:29.460 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:29.460 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:29.567 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T02:27:28.504775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651091112,"num_objects":79485,"num_object_clones":0,"num_object_copies":317940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79512,"num_write_kb":635978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79539,"ondisk_log_size":79539,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2102000,"kb_used_data":1151248,"kb_used_omap":14,"kb_used_meta":950705,"kb_avail":375385360,"statfs":{"total":386547056640,"available":384394608640,"internally_reserved":0,"allocated":1178877952,"data_stored":1178137464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":973522483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,2,8,18,12,15,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30883840,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":30160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003568"},"pg_stats":[{"pgid":"1.f","version":"34'4868","reported_seq":4910,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.510232+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:26.510232+0000","last_peered":"2024-04-26T02:27:26.510232+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:26.510232+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39878656,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":38944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'4999","reported_seq":5065,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:25.082729+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:25.082729+0000","last_peered":"2024-04-26T02:27:25.082729+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:25.082729+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40915468,"num_objects":4995,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4997,"num_write_kb":39972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4921","reported_seq":4999,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.211296+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:28.211296+0000","last_peered":"2024-04-26T02:27:28.211296+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:28.211296+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40286470,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4920,"num_write_kb":39350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4814","reported_seq":4877,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:25.127476+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:25.127476+0000","last_peered":"2024-04-26T02:27:25.127476+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:25.127476+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399948,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":38485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4895","reported_seq":4951,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.538164+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:26.538164+0000","last_peered":"2024-04-26T02:27:26.538164+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:26.538164+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40071692,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4941","reported_seq":4985,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.595480+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:26.595480+0000","last_peered":"2024-04-26T02:27:26.595480+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:26.595480+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40476672,"num_objects":4941,"num_object_clones":0,"num_object_copies":19764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4941,"num_write_kb":39528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'4964","reported_seq":9205,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:25.256264+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:25.256264+0000","last_peered":"2024-04-26T02:27:25.256264+0000","last_clean":"2024-04-26T02:27:25.256264+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:25.256264+0000","last_undegraded":"2024-04-26T02:27:25.256264+0000","last_fullsized":"2024-04-26T02:27:25.256264+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:29.876764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40612364,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4962,"num_write_kb":39676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5121","reported_seq":5187,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:25.224805+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:25.224805+0000","last_peered":"2024-04-26T02:27:25.224805+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:25.224805+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906700,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":40940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4959","reported_seq":5016,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.548300+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:26.548300+0000","last_peered":"2024-04-26T02:27:26.548300+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:26.548300+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40551448,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4959","reported_seq":5038,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.221021+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:28.221021+0000","last_peered":"2024-04-26T02:27:28.221021+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:28.221021+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604172,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4970","reported_seq":5028,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.496124+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:26.496124+0000","last_peered":"2024-04-26T02:27:26.496124+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:26.496124+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40667922,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4967,"num_write_kb":39724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4926","reported_seq":4965,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.577015+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:26.577015+0000","last_peered":"2024-04-26T02:27:26.577015+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:26.577015+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40343814,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4925,"num_write_kb":39399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5000","reported_seq":5074,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331070+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:28.331070+0000","last_peered":"2024-04-26T02:27:28.331070+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:28.331070+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40950022,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4999,"num_write_kb":39991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5163","reported_seq":5216,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.437891+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:26.437891+0000","last_peered":"2024-04-26T02:27:26.437891+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:26.437891+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42258956,"num_objects":5159,"num_object_clones":0,"num_object_copies":20636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5058","reported_seq":6240,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331461+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:28.331461+0000","last_peered":"2024-04-26T02:27:28.331461+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:28.331461+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41406988,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4513792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'4981","reported_seq":5046,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:25.089361+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:25.089361+0000","last_peered":"2024-04-26T02:27:25.089361+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:25.089361+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40759820,"num_objects":4976,"num_object_clones":0,"num_object_copies":19904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":39820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":651091112,"num_objects":79485,"num_object_clones":0,"num_object_copies":317940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79512,"num_write_kb":635978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1179996160,"data_stored":1179996160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79539,"ondisk_log_size":79539,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570092,"kb_used_data":317668,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93801748,"statfs":{"total":96636764160,"available":96052989952,"internally_reserved":0,"allocated":325292032,"data_stored":325106910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,0,3,6,6,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":319000,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":326656000,"data_stored":326470878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396628,"kb_used_data":198412,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93975212,"statfs":{"total":96636764160,"available":96230617088,"internally_reserved":0,"allocated":203173888,"data_stored":202988766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564368,"kb_used_data":316168,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93807472,"statfs":{"total":96636764160,"available":96058851328,"internally_reserved":0,"allocated":323756032,"data_stored":323570910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323944448,"data_stored":323944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202973184,"data_stored":202973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327516160,"data_stored":327516160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325562368,"data_stored":325562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:29.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:29.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 188 20 80343 80323 3.3365 2.17188 0.0717234 0.0468153 2024-04-26T02:27:30.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 189 20 80620 80600 3.33028 2.16406 0.0690412 0.0469017 2024-04-26T02:27:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:30.978 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:31.073 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-04-26T02:27:30.505303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654785704,"num_objects":79936,"num_object_clones":0,"num_object_copies":319744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79963,"num_write_kb":639586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79990,"ondisk_log_size":79990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2111724,"kb_used_data":1156812,"kb_used_omap":14,"kb_used_meta":954865,"kb_avail":375375636,"statfs":{"total":386547056640,"available":384384651264,"internally_reserved":0,"allocated":1184575488,"data_stored":1183835000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":977782323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,4,10,12,10,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26877952,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003576"},"pg_stats":[{"pgid":"1.f","version":"34'4868","reported_seq":4910,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.510232+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:26.510232+0000","last_peered":"2024-04-26T02:27:26.510232+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:26.510232+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39878656,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":38944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5087","reported_seq":5153,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.332153+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:30.332153+0000","last_peered":"2024-04-26T02:27:30.332153+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:30.332153+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41636364,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4921","reported_seq":4999,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.211296+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:28.211296+0000","last_peered":"2024-04-26T02:27:28.211296+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:28.211296+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40286470,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4920,"num_write_kb":39350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4904","reported_seq":4966,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.296905+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:30.296905+0000","last_peered":"2024-04-26T02:27:30.296905+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:30.296905+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40137228,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":39205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4895","reported_seq":4951,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.538164+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:26.538164+0000","last_peered":"2024-04-26T02:27:26.538164+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:26.538164+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40071692,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'4941","reported_seq":4985,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.595480+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:26.595480+0000","last_peered":"2024-04-26T02:27:26.595480+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:26.595480+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40476672,"num_objects":4941,"num_object_clones":0,"num_object_copies":19764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4941,"num_write_kb":39528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5060","reported_seq":9301,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:30.325123+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:30.325123+0000","last_peered":"2024-04-26T02:27:30.325123+0000","last_clean":"2024-04-26T02:27:30.325123+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:30.325123+0000","last_undegraded":"2024-04-26T02:27:30.325123+0000","last_fullsized":"2024-04-26T02:27:30.325123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41398796,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5058,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5207","reported_seq":5273,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.016809+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:30.016809+0000","last_peered":"2024-04-26T02:27:30.016809+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:30.016809+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42611212,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'4959","reported_seq":5016,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.548300+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:26.548300+0000","last_peered":"2024-04-26T02:27:26.548300+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:26.548300+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40551448,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4959","reported_seq":5038,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.221021+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:28.221021+0000","last_peered":"2024-04-26T02:27:28.221021+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:28.221021+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604172,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'4970","reported_seq":5028,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.496124+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:26.496124+0000","last_peered":"2024-04-26T02:27:26.496124+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:26.496124+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40667922,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4967,"num_write_kb":39724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'4926","reported_seq":4965,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.577015+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:26.577015+0000","last_peered":"2024-04-26T02:27:26.577015+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:26.577015+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40343814,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4925,"num_write_kb":39399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5000","reported_seq":5074,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331070+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:28.331070+0000","last_peered":"2024-04-26T02:27:28.331070+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:28.331070+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40950022,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4999,"num_write_kb":39991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5163","reported_seq":5216,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:26.437891+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:26.437891+0000","last_peered":"2024-04-26T02:27:26.437891+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:26.437891+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42258956,"num_objects":5159,"num_object_clones":0,"num_object_copies":20636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5058","reported_seq":6240,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331461+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:28.331461+0000","last_peered":"2024-04-26T02:27:28.331461+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:28.331461+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41406988,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4513792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5072","reported_seq":5137,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.283581+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:30.283581+0000","last_peered":"2024-04-26T02:27:30.283581+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:30.283581+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505292,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654785704,"num_objects":79936,"num_object_clones":0,"num_object_copies":319744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79963,"num_write_kb":639586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1185808384,"data_stored":1185808384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79990,"ondisk_log_size":79990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570092,"kb_used_data":317668,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93801748,"statfs":{"total":96636764160,"available":96052989952,"internally_reserved":0,"allocated":325292032,"data_stored":325106910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,0,3,6,6,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":319000,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":326656000,"data_stored":326470878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396628,"kb_used_data":198412,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93975212,"statfs":{"total":96636764160,"available":96230617088,"internally_reserved":0,"allocated":203173888,"data_stored":202988766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574092,"kb_used_data":321732,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93797748,"statfs":{"total":96636764160,"available":96048893952,"internally_reserved":0,"allocated":329453568,"data_stored":329268446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329756672,"data_stored":329756672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202973184,"data_stored":202973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327516160,"data_stored":327516160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325562368,"data_stored":325562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:31.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:31.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'4868", 'reported_seq': 4910, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.510232+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:26.510232+0000', 'last_peered': '2024-04-26T02:27:26.510232+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:26.510232+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4868, 'log_dups_size': 0, 'ondisk_log_size': 4868, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39878656, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4868, 'num_write_kb': 38944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2851}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:31.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5087", 'reported_seq': 5153, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:30.332153+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:30.332153+0000', 'last_peered': '2024-04-26T02:27:30.332153+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:30.332153+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5087, 'log_dups_size': 0, 'ondisk_log_size': 5087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41636364, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5085, 'num_write_kb': 40676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'4921", 'reported_seq': 4999, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:28.211296+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:28.211296+0000', 'last_peered': '2024-04-26T02:27:28.211296+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:28.211296+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4921, 'log_dups_size': 0, 'ondisk_log_size': 4921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40286470, 'num_objects': 4918, 'num_object_clones': 0, 'num_object_copies': 19672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4920, 'num_write_kb': 39350, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2894}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4904", 'reported_seq': 4966, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:30.296905+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:30.296905+0000', 'last_peered': '2024-04-26T02:27:30.296905+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:30.296905+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4904, 'log_dups_size': 0, 'ondisk_log_size': 4904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40137228, 'num_objects': 4900, 'num_object_clones': 0, 'num_object_copies': 19600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4902, 'num_write_kb': 39205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:31.076 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'4895", 'reported_seq': 4951, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.538164+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:26.538164+0000', 'last_peered': '2024-04-26T02:27:26.538164+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:26.538164+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4895, 'log_dups_size': 0, 'ondisk_log_size': 4895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40071692, 'num_objects': 4892, 'num_object_clones': 0, 'num_object_copies': 19568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4893, 'num_write_kb': 39141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:31.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'4941", 'reported_seq': 4985, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.595480+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:26.595480+0000', 'last_peered': '2024-04-26T02:27:26.595480+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:26.595480+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4941, 'log_dups_size': 0, 'ondisk_log_size': 4941, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40476672, 'num_objects': 4941, 'num_object_clones': 0, 'num_object_copies': 19764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4941, 'num_write_kb': 39528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:31.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5207", 'reported_seq': 5273, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:30.016809+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:30.016809+0000', 'last_peered': '2024-04-26T02:27:30.016809+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:30.016809+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5207, 'log_dups_size': 0, 'ondisk_log_size': 5207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42611212, 'num_objects': 5202, 'num_object_clones': 0, 'num_object_copies': 20808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5205, 'num_write_kb': 41628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'4959", 'reported_seq': 5016, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.548300+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:26.548300+0000', 'last_peered': '2024-04-26T02:27:26.548300+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:26.548300+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4959, 'log_dups_size': 0, 'ondisk_log_size': 4959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40551448, 'num_objects': 4951, 'num_object_clones': 0, 'num_object_copies': 19804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4956, 'num_write_kb': 39625, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2947}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'4959", 'reported_seq': 5038, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:28.221021+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:28.221021+0000', 'last_peered': '2024-04-26T02:27:28.221021+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:28.221021+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4959, 'log_dups_size': 0, 'ondisk_log_size': 4959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40604172, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4957, 'num_write_kb': 39654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2920}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'4970", 'reported_seq': 5028, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.496124+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:26.496124+0000', 'last_peered': '2024-04-26T02:27:26.496124+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:26.496124+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4970, 'log_dups_size': 0, 'ondisk_log_size': 4970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40667922, 'num_objects': 4965, 'num_object_clones': 0, 'num_object_copies': 19860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4967, 'num_write_kb': 39724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:31.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'4926", 'reported_seq': 4965, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.577015+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:26.577015+0000', 'last_peered': '2024-04-26T02:27:26.577015+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:26.577015+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4926, 'log_dups_size': 0, 'ondisk_log_size': 4926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40343814, 'num_objects': 4925, 'num_object_clones': 0, 'num_object_copies': 19700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4925, 'num_write_kb': 39399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5000", 'reported_seq': 5074, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:28.331070+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:28.331070+0000', 'last_peered': '2024-04-26T02:27:28.331070+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:28.331070+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5000, 'log_dups_size': 0, 'ondisk_log_size': 5000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40950022, 'num_objects': 4999, 'num_object_clones': 0, 'num_object_copies': 19996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4999, 'num_write_kb': 39991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5163", 'reported_seq': 5216, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:26.437891+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:26.437891+0000', 'last_peered': '2024-04-26T02:27:26.437891+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:26.437891+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5163, 'log_dups_size': 0, 'ondisk_log_size': 5163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42258956, 'num_objects': 5159, 'num_object_clones': 0, 'num_object_copies': 20636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5161, 'num_write_kb': 41284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:31.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5058", 'reported_seq': 6240, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:28.331461+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:28.331461+0000', 'last_peered': '2024-04-26T02:27:28.331461+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:28.331461+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5058, 'log_dups_size': 0, 'ondisk_log_size': 5058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41406988, 'num_objects': 5055, 'num_object_clones': 0, 'num_object_copies': 20220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1502, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5056, 'num_write_kb': 40445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 553, 'num_bytes_recovered': 4513792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:31.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:31.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5072", 'reported_seq': 5137, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:30.283581+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:30.283581+0000', 'last_peered': '2024-04-26T02:27:30.283581+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:30.283581+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5072, 'log_dups_size': 0, 'ondisk_log_size': 5072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41505292, 'num_objects': 5067, 'num_object_clones': 0, 'num_object_copies': 20268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5070, 'num_write_kb': 40548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:31.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:31.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 190 20 80865 80845 3.32283 1.91406 0.0167429 0.0469735 2024-04-26T02:27:32.110 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:32.111 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30105/319744 objects degraded (9.415%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":248,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79936,"data_bytes":654785704,"bytes_used":2162405376,"bytes_avail":384384651264,"bytes_total":386547056640,"degraded_objects":30105,"degraded_total":319744,"degraded_ratio":0.094153447758206568,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":80530,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"write_bytes_sec":2239161,"read_op_per_sec":0,"write_op_per_sec":273},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:27:32.205 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:32.205 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:32.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 191 20 81123 81103 3.31599 2.01562 0.0503438 0.0471002 2024-04-26T02:27:33.594 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:33.595 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:33.676 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-04-26T02:27:32.506562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659659944,"num_objects":80531,"num_object_clones":0,"num_object_copies":322124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80558,"num_write_kb":644346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80585,"ondisk_log_size":80585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2131272,"kb_used_data":1168040,"kb_used_omap":14,"kb_used_meta":963185,"kb_avail":375356088,"statfs":{"total":386547056640,"available":384364634112,"internally_reserved":0,"allocated":1196072960,"data_stored":1195332472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":986302003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,6,7,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":27408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004407"},"pg_stats":[{"pgid":"1.f","version":"34'4958","reported_seq":4999,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.598144+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:31.598144+0000","last_peered":"2024-04-26T02:27:31.598144+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:31.598144+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40615936,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4958,"num_write_kb":39664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5087","reported_seq":5153,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.332153+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:30.332153+0000","last_peered":"2024-04-26T02:27:30.332153+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:30.332153+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41636364,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'4921","reported_seq":4999,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.211296+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:28.211296+0000","last_peered":"2024-04-26T02:27:28.211296+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:28.211296+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40286470,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4920,"num_write_kb":39350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4904","reported_seq":4966,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.296905+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:30.296905+0000","last_peered":"2024-04-26T02:27:30.296905+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:30.296905+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40137228,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":39205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4974","reported_seq":5030,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.414092+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:31.414092+0000","last_peered":"2024-04-26T02:27:31.414092+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:31.414092+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40718860,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":39773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5021","reported_seq":5064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.529214+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:31.529214+0000","last_peered":"2024-04-26T02:27:31.529214+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:31.529214+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41132032,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5060","reported_seq":9301,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:30.325123+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:30.325123+0000","last_peered":"2024-04-26T02:27:30.325123+0000","last_clean":"2024-04-26T02:27:30.325123+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:30.325123+0000","last_undegraded":"2024-04-26T02:27:30.325123+0000","last_fullsized":"2024-04-26T02:27:30.325123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41398796,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5058,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5207","reported_seq":5273,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.016809+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:30.016809+0000","last_peered":"2024-04-26T02:27:30.016809+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:30.016809+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42611212,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5038","reported_seq":5095,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.473974+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:31.473974+0000","last_peered":"2024-04-26T02:27:31.473974+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:31.473974+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41198616,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'4959","reported_seq":5038,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.221021+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:28.221021+0000","last_peered":"2024-04-26T02:27:28.221021+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:28.221021+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604172,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":39654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5058","reported_seq":5116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.566464+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:31.566464+0000","last_peered":"2024-04-26T02:27:31.566464+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:31.566464+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41388818,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":40428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5022","reported_seq":5061,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.562764+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:31.562764+0000","last_peered":"2024-04-26T02:27:31.562764+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:31.562764+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41130246,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5000","reported_seq":5074,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331070+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:28.331070+0000","last_peered":"2024-04-26T02:27:28.331070+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:28.331070+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40950022,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4999,"num_write_kb":39991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5246","reported_seq":5299,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.534654+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:31.534654+0000","last_peered":"2024-04-26T02:27:31.534654+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:31.534654+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5246,"log_dups_size":0,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42938892,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3171}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5058","reported_seq":6240,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:28.331461+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:28.331461+0000","last_peered":"2024-04-26T02:27:28.331461+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:28.331461+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41406988,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4513792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5072","reported_seq":5137,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:30.283581+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:30.283581+0000","last_peered":"2024-04-26T02:27:30.283581+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:30.283581+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505292,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":40548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":659659944,"num_objects":80531,"num_object_clones":0,"num_object_copies":322124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80558,"num_write_kb":644346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":21569536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197293568,"data_stored":1197293568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80585,"ondisk_log_size":80585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579804,"kb_used_data":323220,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93792036,"statfs":{"total":96636764160,"available":96043044864,"internally_reserved":0,"allocated":330977280,"data_stored":330792158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570912,"kb_used_data":319000,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93800928,"statfs":{"total":96636764160,"available":96052150272,"internally_reserved":0,"allocated":326656000,"data_stored":326470878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406464,"kb_used_data":204088,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93965376,"statfs":{"total":96636764160,"available":96220545024,"internally_reserved":0,"allocated":208986112,"data_stored":208800990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574092,"kb_used_data":321732,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93797748,"statfs":{"total":96636764160,"available":96048893952,"internally_reserved":0,"allocated":329453568,"data_stored":329268446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329756672,"data_stored":329756672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208822272,"data_stored":208822272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327516160,"data_stored":327516160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331198464,"data_stored":331198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:33.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 192 20 81394 81374 3.30974 2.11719 0.0533977 0.0471974 2024-04-26T02:27:34.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 193 20 81644 81624 3.30272 1.95312 0.0623345 0.0472965 2024-04-26T02:27:35.757 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 194 20 81914 81894 3.29657 2.10938 0.101029 0.0473826 2024-04-26T02:27:36.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:36.758 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 195 20 82198 82178 3.29104 2.21875 0.063131 0.0474613 2024-04-26T02:27:37.760 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 196 20 82465 82445 3.28485 2.08594 0.0602538 0.0475479 2024-04-26T02:27:38.116 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:38.116 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:38.240 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-04-26T02:27:36.507470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665705640,"num_objects":81269,"num_object_clones":0,"num_object_copies":325076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81296,"num_write_kb":650250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2741,"num_bytes_recovered":22216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81323,"ondisk_log_size":81323,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2187260,"kb_used_data":1179612,"kb_used_omap":14,"kb_used_meta":1007601,"kb_avail":375300100,"statfs":{"total":386547056640,"available":384307302400,"internally_reserved":0,"allocated":1207922688,"data_stored":1207182200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1031783987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,15,4,2,6,8,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25780224,"num_objects":3147,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":25176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"1.f","version":"34'4958","reported_seq":4999,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.598144+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:31.598144+0000","last_peered":"2024-04-26T02:27:31.598144+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:31.598144+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40615936,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4958,"num_write_kb":39664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5165","reported_seq":5231,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.313559+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:35.313559+0000","last_peered":"2024-04-26T02:27:35.313559+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:35.313559+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42275340,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5163,"num_write_kb":41300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5007","reported_seq":5085,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:33.309917+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:33.309917+0000","last_peered":"2024-04-26T02:27:33.309917+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:33.309917+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40990982,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5006,"num_write_kb":40038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2980}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4986","reported_seq":5049,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.335142+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:35.335142+0000","last_peered":"2024-04-26T02:27:35.335142+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:35.335142+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808972,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'4974","reported_seq":5030,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.414092+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:31.414092+0000","last_peered":"2024-04-26T02:27:31.414092+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:31.414092+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40718860,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":39773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5021","reported_seq":5064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.529214+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:31.529214+0000","last_peered":"2024-04-26T02:27:31.529214+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:31.529214+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41132032,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5142","reported_seq":9384,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:35.266775+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:35.266775+0000","last_peered":"2024-04-26T02:27:35.266775+0000","last_clean":"2024-04-26T02:27:35.266775+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:35.266775+0000","last_undegraded":"2024-04-26T02:27:35.266775+0000","last_fullsized":"2024-04-26T02:27:35.266775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:35.865566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42070540,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":41100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5281","reported_seq":5347,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.305906+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:35.305906+0000","last_peered":"2024-04-26T02:27:35.305906+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:35.305906+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43217420,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5038","reported_seq":5095,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.473974+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:31.473974+0000","last_peered":"2024-04-26T02:27:31.473974+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:31.473974+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41198616,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5037","reported_seq":5116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:33.210937+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:33.210937+0000","last_peered":"2024-04-26T02:27:33.210937+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:33.210937+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243148,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5058","reported_seq":5116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.566464+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:31.566464+0000","last_peered":"2024-04-26T02:27:31.566464+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:31.566464+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41388818,"num_objects":5053,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":40428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5022","reported_seq":5061,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.562764+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:31.562764+0000","last_peered":"2024-04-26T02:27:31.562764+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:31.562764+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41130246,"num_objects":5021,"num_object_clones":0,"num_object_copies":20084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5087","reported_seq":5161,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:33.297198+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:33.297198+0000","last_peered":"2024-04-26T02:27:33.297198+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:33.297198+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41662726,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5246","reported_seq":5299,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:31.534654+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:31.534654+0000","last_peered":"2024-04-26T02:27:31.534654+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:31.534654+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5246,"log_dups_size":0,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42938892,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3171}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5139","reported_seq":6478,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:33.411369+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:33.411369+0000","last_peered":"2024-04-26T02:27:33.411369+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:33.411369+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42070540,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5137,"num_write_kb":41093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5162","reported_seq":5227,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.309212+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:35.309212+0000","last_peered":"2024-04-26T02:27:35.309212+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:35.309212+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42242572,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":41268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":665705640,"num_objects":81269,"num_object_clones":0,"num_object_copies":325076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81296,"num_write_kb":650250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2741,"num_bytes_recovered":22216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208135680,"data_stored":1208135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81323,"ondisk_log_size":81323,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579804,"kb_used_data":323220,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93792036,"statfs":{"total":96636764160,"available":96043044864,"internally_reserved":0,"allocated":330977280,"data_stored":330792158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599500,"kb_used_data":325380,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93772340,"statfs":{"total":96636764160,"available":96022876160,"internally_reserved":0,"allocated":333189120,"data_stored":333003998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,0,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406464,"kb_used_data":204088,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93965376,"statfs":{"total":96636764160,"available":96220545024,"internally_reserved":0,"allocated":208986112,"data_stored":208800990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601492,"kb_used_data":326924,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93770348,"statfs":{"total":96636764160,"available":96020836352,"internally_reserved":0,"allocated":334770176,"data_stored":334585054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,1,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335142912,"data_stored":335142912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208822272,"data_stored":208822272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332972032,"data_stored":332972032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331198464,"data_stored":331198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:38.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:38.761 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 197 20 82729 82709 3.27865 2.0625 0.145916 0.047642 2024-04-26T02:27:39.632 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:39.632 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:39.718 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-04-26T02:27:38.508016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":673242280,"num_objects":82189,"num_object_clones":0,"num_object_copies":328756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82216,"num_write_kb":657610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":22781952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82243,"ondisk_log_size":82243,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2210400,"kb_used_data":1195776,"kb_used_omap":14,"kb_used_meta":1014577,"kb_avail":375276960,"statfs":{"total":386547056640,"available":384283607040,"internally_reserved":0,"allocated":1224474624,"data_stored":1223734136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1038927411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,7,4,9,10,6,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29900800,"num_objects":3650,"num_object_clones":0,"num_object_copies":14600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":29200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"1.f","version":"34'5039","reported_seq":5080,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.542525+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:36.542525+0000","last_peered":"2024-04-26T02:27:36.542525+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:36.542525+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41279488,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":40312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5165","reported_seq":5231,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.313559+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:35.313559+0000","last_peered":"2024-04-26T02:27:35.313559+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:35.313559+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42275340,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5163,"num_write_kb":41300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5091","reported_seq":5169,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.328675+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:38.328675+0000","last_peered":"2024-04-26T02:27:38.328675+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:38.328675+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41679110,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5090,"num_write_kb":40710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3064}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'4986","reported_seq":5049,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.335142+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:35.335142+0000","last_peered":"2024-04-26T02:27:35.335142+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:35.335142+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808972,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5059","reported_seq":5115,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.605338+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:36.605338+0000","last_peered":"2024-04-26T02:27:36.605338+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:36.605338+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5059,"log_dups_size":0,"ondisk_log_size":5059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415180,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5057,"num_write_kb":40453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3045}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5098","reported_seq":5142,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.650244+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:36.650244+0000","last_peered":"2024-04-26T02:27:36.650244+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:36.650244+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5098,"log_dups_size":0,"ondisk_log_size":5098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41762816,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5098,"num_write_kb":40784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5142","reported_seq":9384,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:35.266775+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:35.266775+0000","last_peered":"2024-04-26T02:27:35.266775+0000","last_clean":"2024-04-26T02:27:35.266775+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:35.266775+0000","last_undegraded":"2024-04-26T02:27:35.266775+0000","last_fullsized":"2024-04-26T02:27:35.266775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:35.865566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42070540,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":41100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5281","reported_seq":5347,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.305906+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:35.305906+0000","last_peered":"2024-04-26T02:27:35.305906+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:35.305906+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43217420,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5127","reported_seq":5184,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.589585+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:36.589585+0000","last_peered":"2024-04-26T02:27:36.589585+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:36.589585+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41927704,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5119","reported_seq":5198,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.372354+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:38.372354+0000","last_peered":"2024-04-26T02:27:38.372354+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:38.372354+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41914892,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":40934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5134","reported_seq":5192,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.520450+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:36.520450+0000","last_peered":"2024-04-26T02:27:36.520450+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:36.520450+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42011410,"num_objects":5129,"num_object_clones":0,"num_object_copies":20516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5131,"num_write_kb":41036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5109","reported_seq":5148,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.672283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:36.672283+0000","last_peered":"2024-04-26T02:27:36.672283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:36.672283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41842950,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3120}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5171","reported_seq":5246,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.263780+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:38.263780+0000","last_peered":"2024-04-26T02:27:38.263780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:38.263780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42350854,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5170,"num_write_kb":41359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5322","reported_seq":5375,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:36.585678+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:36.585678+0000","last_peered":"2024-04-26T02:27:36.585678+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:36.585678+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5322,"log_dups_size":0,"ondisk_log_size":5322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43561484,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5320,"num_write_kb":42556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3247}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5238","reported_seq":6716,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.376896+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:38.376896+0000","last_peered":"2024-04-26T02:27:38.376896+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:38.376896+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42881548,"num_objects":5235,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":41885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":701,"num_bytes_recovered":5726208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5162","reported_seq":5227,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:35.309212+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:35.309212+0000","last_peered":"2024-04-26T02:27:35.309212+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:35.309212+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42242572,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":41268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":673242280,"num_objects":82189,"num_object_clones":0,"num_object_copies":328756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82216,"num_write_kb":657610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":22781952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1224925184,"data_stored":1224925184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82243,"ondisk_log_size":82243,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594632,"kb_used_data":328320,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93777208,"statfs":{"total":96636764160,"available":96027860992,"internally_reserved":0,"allocated":336199680,"data_stored":336014558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":272690896},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,6,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614660,"kb_used_data":330812,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93757180,"statfs":{"total":96636764160,"available":96007352320,"internally_reserved":0,"allocated":338751488,"data_stored":338566366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399616,"kb_used_data":209720,"kb_used_omap":4,"kb_used_meta":189883,"kb_avail":93972224,"statfs":{"total":96636764160,"available":96227557376,"internally_reserved":0,"allocated":214753280,"data_stored":214568158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194440911},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601492,"kb_used_data":326924,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93770348,"statfs":{"total":96636764160,"available":96020836352,"internally_reserved":0,"allocated":334770176,"data_stored":334585054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,1,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335142912,"data_stored":335142912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":214716416,"data_stored":214716416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":338477056,"data_stored":338477056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336588800,"data_stored":336588800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:39.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:39.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5039", 'reported_seq': 5080, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.542525+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:36.542525+0000', 'last_peered': '2024-04-26T02:27:36.542525+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:36.542525+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5039, 'log_dups_size': 0, 'ondisk_log_size': 5039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41279488, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5039, 'num_write_kb': 40312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3022}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5165", 'reported_seq': 5231, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:35.313559+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:35.313559+0000', 'last_peered': '2024-04-26T02:27:35.313559+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:35.313559+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5165, 'log_dups_size': 0, 'ondisk_log_size': 5165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42275340, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 20644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5163, 'num_write_kb': 41300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5091", 'reported_seq': 5169, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:38.328675+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:38.328675+0000', 'last_peered': '2024-04-26T02:27:38.328675+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:38.328675+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5091, 'log_dups_size': 0, 'ondisk_log_size': 5091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41679110, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5090, 'num_write_kb': 40710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3064}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'4986", 'reported_seq': 5049, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:35.335142+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:35.335142+0000', 'last_peered': '2024-04-26T02:27:35.335142+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:35.335142+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 4986, 'log_dups_size': 0, 'ondisk_log_size': 4986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40808972, 'num_objects': 4982, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4984, 'num_write_kb': 39861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5059", 'reported_seq': 5115, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.605338+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:36.605338+0000', 'last_peered': '2024-04-26T02:27:36.605338+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:36.605338+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5059, 'log_dups_size': 0, 'ondisk_log_size': 5059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41415180, 'num_objects': 5056, 'num_object_clones': 0, 'num_object_copies': 20224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5057, 'num_write_kb': 40453, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3045}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5098", 'reported_seq': 5142, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.650244+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:36.650244+0000', 'last_peered': '2024-04-26T02:27:36.650244+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:36.650244+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5098, 'log_dups_size': 0, 'ondisk_log_size': 5098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41762816, 'num_objects': 5098, 'num_object_clones': 0, 'num_object_copies': 20392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5098, 'num_write_kb': 40784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5281", 'reported_seq': 5347, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:35.305906+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:35.305906+0000', 'last_peered': '2024-04-26T02:27:35.305906+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:35.305906+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5281, 'log_dups_size': 0, 'ondisk_log_size': 5281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43217420, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5279, 'num_write_kb': 42220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5127", 'reported_seq': 5184, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.589585+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:36.589585+0000', 'last_peered': '2024-04-26T02:27:36.589585+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:36.589585+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5127, 'log_dups_size': 0, 'ondisk_log_size': 5127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41927704, 'num_objects': 5119, 'num_object_clones': 0, 'num_object_copies': 20476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5124, 'num_write_kb': 40969, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3115}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5119", 'reported_seq': 5198, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:38.372354+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:38.372354+0000', 'last_peered': '2024-04-26T02:27:38.372354+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:38.372354+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5119, 'log_dups_size': 0, 'ondisk_log_size': 5119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41914892, 'num_objects': 5117, 'num_object_clones': 0, 'num_object_copies': 20468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5117, 'num_write_kb': 40934, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3080}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5134", 'reported_seq': 5192, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.520450+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:36.520450+0000', 'last_peered': '2024-04-26T02:27:36.520450+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:36.520450+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5134, 'log_dups_size': 0, 'ondisk_log_size': 5134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42011410, 'num_objects': 5129, 'num_object_clones': 0, 'num_object_copies': 20516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5131, 'num_write_kb': 41036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5109", 'reported_seq': 5148, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.672283+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:36.672283+0000', 'last_peered': '2024-04-26T02:27:36.672283+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:36.672283+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5109, 'log_dups_size': 0, 'ondisk_log_size': 5109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41842950, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 20432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5108, 'num_write_kb': 40863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:39.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5171", 'reported_seq': 5246, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:38.263780+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:38.263780+0000', 'last_peered': '2024-04-26T02:27:38.263780+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:38.263780+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5171, 'log_dups_size': 0, 'ondisk_log_size': 5171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42350854, 'num_objects': 5170, 'num_object_clones': 0, 'num_object_copies': 20680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5170, 'num_write_kb': 41359, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5322", 'reported_seq': 5375, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:36.585678+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:36.585678+0000', 'last_peered': '2024-04-26T02:27:36.585678+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:36.585678+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5322, 'log_dups_size': 0, 'ondisk_log_size': 5322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43561484, 'num_objects': 5318, 'num_object_clones': 0, 'num_object_copies': 21272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5320, 'num_write_kb': 42556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5238", 'reported_seq': 6716, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:38.376896+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:38.376896+0000', 'last_peered': '2024-04-26T02:27:38.376896+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:38.376896+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5238, 'log_dups_size': 0, 'ondisk_log_size': 5238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42881548, 'num_objects': 5235, 'num_object_clones': 0, 'num_object_copies': 20940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1354, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5236, 'num_write_kb': 41885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 701, 'num_bytes_recovered': 5726208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:39.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5162", 'reported_seq': 5227, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:35.309212+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:35.309212+0000', 'last_peered': '2024-04-26T02:27:35.309212+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:35.309212+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5162, 'log_dups_size': 0, 'ondisk_log_size': 5162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42242572, 'num_objects': 5157, 'num_object_clones': 0, 'num_object_copies': 20628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5160, 'num_write_kb': 41268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:39.722 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:39.763 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 198 20 82999 82979 3.2727 2.10938 0.0188263 0.0477292 2024-04-26T02:27:40.766 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 199 20 83261 83241 3.26651 2.04688 0.0647471 0.0478147 2024-04-26T02:27:41.765 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:27:41.765710+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0479004 2024-04-26T02:27:41.765 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:27:41.765 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 200 20 83532 83512 3.26076 2.11719 0.0709188 0.0479004 2024-04-26T02:27:41.824 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:41.824 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29957/330440 objects degraded (9.066%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82610,"data_bytes":676691112,"bytes_used":2278961152,"bytes_avail":384268095488,"bytes_total":386547056640,"degraded_objects":29957,"degraded_total":330440,"degraded_ratio":0.09065791066456845,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":101002,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"write_bytes_sec":2132653,"read_op_per_sec":0,"write_op_per_sec":260},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:27:41.901 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:41.901 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:42.767 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 201 20 83795 83775 3.25473 2.05469 0.018452 0.0479879 2024-04-26T02:27:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:43.255 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:43.390 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-26T02:27:42.509645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681696424,"num_objects":83221,"num_object_clones":0,"num_object_copies":332884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83248,"num_write_kb":665866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":22781952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83275,"ondisk_log_size":83275,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2267032,"kb_used_data":1212152,"kb_used_omap":14,"kb_used_meta":1054833,"kb_avail":375220328,"statfs":{"total":386547056640,"available":384225615872,"internally_reserved":0,"allocated":1241243648,"data_stored":1240503160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1080149555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,18,9,12,7,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26910720,"num_objects":3285,"num_object_clones":0,"num_object_copies":13140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3285,"num_write_kb":26280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004342"},"pg_stats":[{"pgid":"1.f","version":"34'5124","reported_seq":5166,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.704393+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:41.704393+0000","last_peered":"2024-04-26T02:27:41.704393+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:41.704393+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41975808,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5258","reported_seq":5324,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:40.349864+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:40.349864+0000","last_peered":"2024-04-26T02:27:40.349864+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:40.349864+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43037196,"num_objects":5254,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":42044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5091","reported_seq":5169,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.328675+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:38.328675+0000","last_peered":"2024-04-26T02:27:38.328675+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:38.328675+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41679110,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5090,"num_write_kb":40710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3064}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5064","reported_seq":5127,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:40.330802+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:40.330802+0000","last_peered":"2024-04-26T02:27:40.330802+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:40.330802+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41447948,"num_objects":5060,"num_object_clones":0,"num_object_copies":20240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5062,"num_write_kb":40485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3092}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5149","reported_seq":5205,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.479529+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:41.479529+0000","last_peered":"2024-04-26T02:27:41.479529+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:41.479529+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5149,"log_dups_size":0,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42152460,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5147,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5170","reported_seq":5214,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.574345+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:41.574345+0000","last_peered":"2024-04-26T02:27:41.574345+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:41.574345+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42352640,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5170,"num_write_kb":41360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5223","reported_seq":9465,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:40.182127+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:40.182127+0000","last_peered":"2024-04-26T02:27:40.182127+0000","last_clean":"2024-04-26T02:27:40.182127+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:40.182127+0000","last_undegraded":"2024-04-26T02:27:40.182127+0000","last_fullsized":"2024-04-26T02:27:40.182127+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:40.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734092,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5366","reported_seq":5432,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:40.309633+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:40.309633+0000","last_peered":"2024-04-26T02:27:40.309633+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:40.309633+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43913740,"num_objects":5361,"num_object_clones":0,"num_object_copies":21444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":42900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5228","reported_seq":5285,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.722235+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:41.722235+0000","last_peered":"2024-04-26T02:27:41.722235+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:41.722235+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42755096,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5119","reported_seq":5198,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.372354+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:38.372354+0000","last_peered":"2024-04-26T02:27:38.372354+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:38.372354+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41914892,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":40934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5219","reported_seq":5277,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.609141+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:41.609141+0000","last_peered":"2024-04-26T02:27:41.609141+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:41.609141+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42707730,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5187","reported_seq":5226,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.654678+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:41.654678+0000","last_peered":"2024-04-26T02:27:41.654678+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:41.654678+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42481926,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":41487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5171","reported_seq":5246,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.263780+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:38.263780+0000","last_peered":"2024-04-26T02:27:38.263780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:38.263780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42350854,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5170,"num_write_kb":41359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3059}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5422","reported_seq":5475,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.710570+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:41.710570+0000","last_peered":"2024-04-26T02:27:41.710570+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:41.710570+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44380684,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5238","reported_seq":6716,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:38.376896+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:38.376896+0000","last_peered":"2024-04-26T02:27:38.376896+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:38.376896+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42881548,"num_objects":5235,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":41885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":701,"num_bytes_recovered":5726208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5246","reported_seq":5311,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:40.312880+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:40.312880+0000","last_peered":"2024-04-26T02:27:40.312880+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:40.312880+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5246,"log_dups_size":0,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42930700,"num_objects":5241,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681696424,"num_objects":83221,"num_object_clones":0,"num_object_copies":332884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83248,"num_write_kb":665866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":22781952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241841664,"data_stored":1241841664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83275,"ondisk_log_size":83275,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622092,"kb_used_data":333572,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93749748,"statfs":{"total":96636764160,"available":95999741952,"internally_reserved":0,"allocated":341577728,"data_stored":341392606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":295431888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614660,"kb_used_data":330812,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93757180,"statfs":{"total":96636764160,"available":96007352320,"internally_reserved":0,"allocated":338751488,"data_stored":338566366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413640,"kb_used_data":215424,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93958200,"statfs":{"total":96636764160,"available":96213196800,"internally_reserved":0,"allocated":220594176,"data_stored":220409054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616640,"kb_used_data":332344,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93755200,"statfs":{"total":96636764160,"available":96005324800,"internally_reserved":0,"allocated":340320256,"data_stored":340135134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,12,1,0,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340660224,"data_stored":340660224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220553216,"data_stored":220553216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":338477056,"data_stored":338477056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342151168,"data_stored":342151168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:43.767 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 202 20 84061 84041 3.24891 2.07812 0.162296 0.0480759 2024-04-26T02:27:44.770 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 203 20 84335 84315 3.24341 2.14062 0.061684 0.0481592 2024-04-26T02:27:45.771 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 204 20 84611 84591 3.23806 2.15625 0.0635406 0.0482367 2024-04-26T02:27:46.392 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:46.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 205 20 84852 84832 3.23141 1.88281 0.0817716 0.0483366 2024-04-26T02:27:47.393 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:47.393 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:47.513 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-04-26T02:27:46.514540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":687840424,"num_objects":83971,"num_object_clones":0,"num_object_copies":335884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83998,"num_write_kb":671866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2873,"num_bytes_recovered":23298048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84025,"ondisk_log_size":84025,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2286024,"kb_used_data":1222824,"kb_used_omap":14,"kb_used_meta":1063153,"kb_avail":375201336,"statfs":{"total":386547056640,"available":384206168064,"internally_reserved":0,"allocated":1252171776,"data_stored":1251431288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1088669235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,14,14,5,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25460736,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007473"},"pg_stats":[{"pgid":"1.f","version":"34'5124","reported_seq":5166,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.704393+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:41.704393+0000","last_peered":"2024-04-26T02:27:41.704393+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:41.704393+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41975808,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":40992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5352","reported_seq":5418,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.445476+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:45.445476+0000","last_peered":"2024-04-26T02:27:45.445476+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:45.445476+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43807244,"num_objects":5348,"num_object_clones":0,"num_object_copies":21392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":42796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5189","reported_seq":5267,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.359926+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:43.359926+0000","last_peered":"2024-04-26T02:27:43.359926+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:43.359926+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42481926,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":41494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3162}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5151","reported_seq":5214,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.383188+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:45.383188+0000","last_peered":"2024-04-26T02:27:45.383188+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:45.383188+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42160652,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":41181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5149","reported_seq":5205,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.479529+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:41.479529+0000","last_peered":"2024-04-26T02:27:41.479529+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:41.479529+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5149,"log_dups_size":0,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42152460,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5147,"num_write_kb":41173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5170","reported_seq":5214,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.574345+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:41.574345+0000","last_peered":"2024-04-26T02:27:41.574345+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:41.574345+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42352640,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5170,"num_write_kb":41360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5304","reported_seq":9546,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:45.426017+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:45.426017+0000","last_peered":"2024-04-26T02:27:45.426017+0000","last_clean":"2024-04-26T02:27:45.426017+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:45.426017+0000","last_undegraded":"2024-04-26T02:27:45.426017+0000","last_fullsized":"2024-04-26T02:27:45.426017+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:46.838111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43397644,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5302,"num_write_kb":42396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5444","reported_seq":5510,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.349554+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:45.349554+0000","last_peered":"2024-04-26T02:27:45.349554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:45.349554+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44552716,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5228","reported_seq":5285,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.722235+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:41.722235+0000","last_peered":"2024-04-26T02:27:41.722235+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:41.722235+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42755096,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":41777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5199","reported_seq":5278,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.322813+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:43.322813+0000","last_peered":"2024-04-26T02:27:43.322813+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:43.322813+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42570252,"num_objects":5197,"num_object_clones":0,"num_object_copies":20788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":41574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5219","reported_seq":5277,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.609141+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:41.609141+0000","last_peered":"2024-04-26T02:27:41.609141+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:41.609141+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42707730,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5187","reported_seq":5226,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.654678+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:41.654678+0000","last_peered":"2024-04-26T02:27:41.654678+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:41.654678+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42481926,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":41487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5252","reported_seq":5327,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.330150+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:43.330150+0000","last_peered":"2024-04-26T02:27:43.330150+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:43.330150+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43014406,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":42007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3140}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5422","reported_seq":5475,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:41.710570+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:41.710570+0000","last_peered":"2024-04-26T02:27:41.710570+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:41.710570+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44380684,"num_objects":5418,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5314","reported_seq":6918,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.445594+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:43.445594+0000","last_peered":"2024-04-26T02:27:43.445594+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:43.445594+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43504140,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":6242304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5321","reported_seq":5386,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.436378+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:45.436378+0000","last_peered":"2024-04-26T02:27:45.436378+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:45.436378+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545100,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":687840424,"num_objects":83971,"num_object_clones":0,"num_object_copies":335884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83998,"num_write_kb":671866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2873,"num_bytes_recovered":23298048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252876288,"data_stored":1252876288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84025,"ondisk_log_size":84025,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622092,"kb_used_data":333572,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93749748,"statfs":{"total":96636764160,"available":95999741952,"internally_reserved":0,"allocated":341577728,"data_stored":341392606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":295431888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,7,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624108,"kb_used_data":336100,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93747732,"statfs":{"total":96636764160,"available":95997677568,"internally_reserved":0,"allocated":344166400,"data_stored":343981278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413640,"kb_used_data":215424,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93958200,"statfs":{"total":96636764160,"available":96213196800,"internally_reserved":0,"allocated":220594176,"data_stored":220409054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626184,"kb_used_data":337728,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93745656,"statfs":{"total":96636764160,"available":95995551744,"internally_reserved":0,"allocated":345833472,"data_stored":345648350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346234880,"data_stored":346234880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220553216,"data_stored":220553216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343937024,"data_stored":343937024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342151168,"data_stored":342151168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:47.514 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:47.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 206 20 85091 85071 3.22479 1.86719 0.0777924 0.0484366 2024-04-26T02:27:48.774 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 207 20 85354 85334 3.21914 2.05469 0.0198914 0.0485141 2024-04-26T02:27:49.074 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:49.079 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:49.158 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-26T02:27:48.516898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692534440,"num_objects":84544,"num_object_clones":0,"num_object_copies":338176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84571,"num_write_kb":676450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2873,"num_bytes_recovered":23298048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84598,"ondisk_log_size":84598,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2305088,"kb_used_data":1233568,"kb_used_omap":14,"kb_used_meta":1071473,"kb_avail":375182272,"statfs":{"total":386547056640,"available":384186646528,"internally_reserved":0,"allocated":1263173632,"data_stored":1262433144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1097188915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,11,4,16,14,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26828800,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3275,"num_write_kb":26200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009427"},"pg_stats":[{"pgid":"1.f","version":"34'5204","reported_seq":5246,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.640598+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:46.640598+0000","last_peered":"2024-04-26T02:27:46.640598+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:46.640598+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42631168,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5352","reported_seq":5418,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.445476+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:45.445476+0000","last_peered":"2024-04-26T02:27:45.445476+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:45.445476+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43807244,"num_objects":5348,"num_object_clones":0,"num_object_copies":21392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":42796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5189","reported_seq":5267,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.359926+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:43.359926+0000","last_peered":"2024-04-26T02:27:43.359926+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:43.359926+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42481926,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":41494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3162}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5151","reported_seq":5214,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.383188+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:45.383188+0000","last_peered":"2024-04-26T02:27:45.383188+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:45.383188+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42160652,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":41181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5245","reported_seq":5301,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.831382+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:46.831382+0000","last_peered":"2024-04-26T02:27:46.831382+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:46.831382+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42938892,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":41941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5232","reported_seq":5275,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.684048+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:46.684048+0000","last_peered":"2024-04-26T02:27:46.684048+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:46.684048+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5232,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5304","reported_seq":9546,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:45.426017+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:45.426017+0000","last_peered":"2024-04-26T02:27:45.426017+0000","last_clean":"2024-04-26T02:27:45.426017+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:45.426017+0000","last_undegraded":"2024-04-26T02:27:45.426017+0000","last_fullsized":"2024-04-26T02:27:45.426017+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:46.838111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43397644,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5302,"num_write_kb":42396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5444","reported_seq":5510,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.349554+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:45.349554+0000","last_peered":"2024-04-26T02:27:45.349554+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:45.349554+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44552716,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5314","reported_seq":5370,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.837511+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:46.837511+0000","last_peered":"2024-04-26T02:27:46.837511+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:46.837511+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43459608,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5311,"num_write_kb":42465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5199","reported_seq":5278,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.322813+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:43.322813+0000","last_peered":"2024-04-26T02:27:43.322813+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:43.322813+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42570252,"num_objects":5197,"num_object_clones":0,"num_object_copies":20788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":41574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5294","reported_seq":5352,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.877507+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:46.877507+0000","last_peered":"2024-04-26T02:27:46.877507+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:46.877507+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43322130,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":42316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5277","reported_seq":5316,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.753597+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:46.753597+0000","last_peered":"2024-04-26T02:27:46.753597+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:46.753597+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43219206,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":42207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5252","reported_seq":5327,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.330150+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:43.330150+0000","last_peered":"2024-04-26T02:27:43.330150+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:43.330150+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43014406,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":42007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3140}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5506","reported_seq":5559,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.674333+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:46.674333+0000","last_peered":"2024-04-26T02:27:46.674333+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:46.674333+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45068812,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5504,"num_write_kb":44028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5314","reported_seq":6918,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:43.445594+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:43.445594+0000","last_peered":"2024-04-26T02:27:43.445594+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:43.445594+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43504140,"num_objects":5311,"num_object_clones":0,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":6242304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5321","reported_seq":5386,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:45.436378+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:45.436378+0000","last_peered":"2024-04-26T02:27:45.436378+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:45.436378+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545100,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692534440,"num_objects":84544,"num_object_clones":0,"num_object_copies":338176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84571,"num_write_kb":676450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2873,"num_bytes_recovered":23298048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1264087040,"data_stored":1264087040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84598,"ondisk_log_size":84598,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631484,"kb_used_data":338804,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93740356,"statfs":{"total":96636764160,"available":95990124544,"internally_reserved":0,"allocated":346935296,"data_stored":346750174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":299691728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,6,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624108,"kb_used_data":336100,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93747732,"statfs":{"total":96636764160,"available":95997677568,"internally_reserved":0,"allocated":344166400,"data_stored":343981278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423312,"kb_used_data":220936,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93948528,"statfs":{"total":96636764160,"available":96203292672,"internally_reserved":0,"allocated":226238464,"data_stored":226053342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,8,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626184,"kb_used_data":337728,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93745656,"statfs":{"total":96636764160,"available":95995551744,"internally_reserved":0,"allocated":345833472,"data_stored":345648350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346234880,"data_stored":346234880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226234368,"data_stored":226234368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343937024,"data_stored":343937024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347680768,"data_stored":347680768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:49.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:49.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5204", 'reported_seq': 5246, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.640598+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:46.640598+0000', 'last_peered': '2024-04-26T02:27:46.640598+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:46.640598+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5204, 'log_dups_size': 0, 'ondisk_log_size': 5204, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42631168, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5204, 'num_write_kb': 41632, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3187}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:49.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5352", 'reported_seq': 5418, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:45.445476+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:45.445476+0000', 'last_peered': '2024-04-26T02:27:45.445476+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:45.445476+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5352, 'log_dups_size': 0, 'ondisk_log_size': 5352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43807244, 'num_objects': 5348, 'num_object_clones': 0, 'num_object_copies': 21392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5350, 'num_write_kb': 42796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5189", 'reported_seq': 5267, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:43.359926+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:43.359926+0000', 'last_peered': '2024-04-26T02:27:43.359926+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:43.359926+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5189, 'log_dups_size': 0, 'ondisk_log_size': 5189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42481926, 'num_objects': 5186, 'num_object_clones': 0, 'num_object_copies': 20744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5188, 'num_write_kb': 41494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3162}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5151", 'reported_seq': 5214, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:45.383188+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:45.383188+0000', 'last_peered': '2024-04-26T02:27:45.383188+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:45.383188+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5151, 'log_dups_size': 0, 'ondisk_log_size': 5151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42160652, 'num_objects': 5147, 'num_object_clones': 0, 'num_object_copies': 20588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5149, 'num_write_kb': 41181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5245", 'reported_seq': 5301, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.831382+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:46.831382+0000', 'last_peered': '2024-04-26T02:27:46.831382+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:46.831382+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5245, 'log_dups_size': 0, 'ondisk_log_size': 5245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42938892, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5243, 'num_write_kb': 41941, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3231}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5232", 'reported_seq': 5275, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.684048+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:46.684048+0000', 'last_peered': '2024-04-26T02:27:46.684048+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:46.684048+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5232, 'log_dups_size': 0, 'ondisk_log_size': 5232, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42860544, 'num_objects': 5232, 'num_object_clones': 0, 'num_object_copies': 20928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5232, 'num_write_kb': 41856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:49.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5444", 'reported_seq': 5510, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:45.349554+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:45.349554+0000', 'last_peered': '2024-04-26T02:27:45.349554+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:45.349554+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5444, 'log_dups_size': 0, 'ondisk_log_size': 5444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44552716, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5442, 'num_write_kb': 43524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5314", 'reported_seq': 5370, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.837511+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:46.837511+0000', 'last_peered': '2024-04-26T02:27:46.837511+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:46.837511+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5314, 'log_dups_size': 0, 'ondisk_log_size': 5314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43459608, 'num_objects': 5306, 'num_object_clones': 0, 'num_object_copies': 21224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5311, 'num_write_kb': 42465, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3302}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5199", 'reported_seq': 5278, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:43.322813+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:43.322813+0000', 'last_peered': '2024-04-26T02:27:43.322813+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:43.322813+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5199, 'log_dups_size': 0, 'ondisk_log_size': 5199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42570252, 'num_objects': 5197, 'num_object_clones': 0, 'num_object_copies': 20788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5197, 'num_write_kb': 41574, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5294", 'reported_seq': 5352, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.877507+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:46.877507+0000', 'last_peered': '2024-04-26T02:27:46.877507+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:46.877507+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5294, 'log_dups_size': 0, 'ondisk_log_size': 5294, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43322130, 'num_objects': 5289, 'num_object_clones': 0, 'num_object_copies': 21156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5291, 'num_write_kb': 42316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3272}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5277", 'reported_seq': 5316, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.753597+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:46.753597+0000', 'last_peered': '2024-04-26T02:27:46.753597+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:46.753597+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5277, 'log_dups_size': 0, 'ondisk_log_size': 5277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43219206, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5276, 'num_write_kb': 42207, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5252", 'reported_seq': 5327, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:43.330150+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:43.330150+0000', 'last_peered': '2024-04-26T02:27:43.330150+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:43.330150+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5252, 'log_dups_size': 0, 'ondisk_log_size': 5252, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43014406, 'num_objects': 5251, 'num_object_clones': 0, 'num_object_copies': 21004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5251, 'num_write_kb': 42007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:49.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:49.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5506", 'reported_seq': 5559, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:46.674333+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:46.674333+0000', 'last_peered': '2024-04-26T02:27:46.674333+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:46.674333+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5506, 'log_dups_size': 0, 'ondisk_log_size': 5506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45068812, 'num_objects': 5502, 'num_object_clones': 0, 'num_object_copies': 22008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5504, 'num_write_kb': 44028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:49.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:49.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5314", 'reported_seq': 6918, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:43.445594+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:43.445594+0000', 'last_peered': '2024-04-26T02:27:43.445594+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:43.445594+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5314, 'log_dups_size': 0, 'ondisk_log_size': 5314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43504140, 'num_objects': 5311, 'num_object_clones': 0, 'num_object_copies': 21244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5312, 'num_write_kb': 42493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 764, 'num_bytes_recovered': 6242304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:49.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:49.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5321", 'reported_seq': 5386, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:45.436378+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:45.436378+0000', 'last_peered': '2024-04-26T02:27:45.436378+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:45.436378+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5321, 'log_dups_size': 0, 'ondisk_log_size': 5321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43545100, 'num_objects': 5316, 'num_object_clones': 0, 'num_object_copies': 21264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 42540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:49.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:49.776 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 208 20 85620 85600 3.21366 2.07812 0.032226 0.0486048 2024-04-26T02:27:50.624 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:50.624 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29894/338176 objects degraded (8.840%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84544,"data_bytes":692534440,"bytes_used":2360410112,"bytes_avail":384186646528,"bytes_total":386547056640,"degraded_objects":29894,"degraded_total":338176,"degraded_ratio":0.088397757380772143,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":90041,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":1081344,"num_keys_recovered":0,"write_bytes_sec":2233978,"read_op_per_sec":0,"write_op_per_sec":272},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:27:50.714 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:27:50.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:50.778 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 209 20 85868 85848 3.20751 1.9375 0.0854581 0.0486991 2024-04-26T02:27:51.782 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 210 20 86135 86115 3.20215 2.08594 0.117078 0.0487777 2024-04-26T02:27:52.434 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:52.435 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:52.477 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-26T02:27:50.517442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":698694824,"num_objects":85296,"num_object_clones":0,"num_object_copies":341184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85323,"num_write_kb":682466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2967,"num_bytes_recovered":24068096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85350,"ondisk_log_size":85350,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2322748,"kb_used_data":1242908,"kb_used_omap":14,"kb_used_meta":1079793,"kb_avail":375164612,"statfs":{"total":386547056640,"available":384168562688,"internally_reserved":0,"allocated":1272737792,"data_stored":1271997304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1105708595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,13,4,21,20,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25452544,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":24856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009425"},"pg_stats":[{"pgid":"1.f","version":"34'5204","reported_seq":5246,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.640598+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:46.640598+0000","last_peered":"2024-04-26T02:27:46.640598+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:46.640598+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42631168,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":41632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5417","reported_seq":5483,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:50.337222+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:50.337222+0000","last_peered":"2024-04-26T02:27:50.337222+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:50.337222+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44339724,"num_objects":5413,"num_object_clones":0,"num_object_copies":21652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5271","reported_seq":5349,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:48.440642+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:48.440642+0000","last_peered":"2024-04-26T02:27:48.440642+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:48.440642+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43153670,"num_objects":5268,"num_object_clones":0,"num_object_copies":21072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3244}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5240","reported_seq":5303,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:50.184273+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:50.184273+0000","last_peered":"2024-04-26T02:27:50.184273+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:50.184273+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42889740,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":41893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5245","reported_seq":5301,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.831382+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:46.831382+0000","last_peered":"2024-04-26T02:27:46.831382+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:46.831382+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42938892,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5243,"num_write_kb":41941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5232","reported_seq":5275,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.684048+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:46.684048+0000","last_peered":"2024-04-26T02:27:46.684048+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:46.684048+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5232,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5391","reported_seq":9632,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:50.372315+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:50.372315+0000","last_peered":"2024-04-26T02:27:50.372315+0000","last_clean":"2024-04-26T02:27:50.372315+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:50.372315+0000","last_undegraded":"2024-04-26T02:27:50.372315+0000","last_fullsized":"2024-04-26T02:27:50.372315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:51.907287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44110348,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":43092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5528","reported_seq":5594,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:50.185421+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:50.185421+0000","last_peered":"2024-04-26T02:27:50.185421+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:50.185421+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45240844,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5526,"num_write_kb":44196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3382}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5314","reported_seq":5370,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.837511+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:46.837511+0000","last_peered":"2024-04-26T02:27:46.837511+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:46.837511+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43459608,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5311,"num_write_kb":42465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5285","reported_seq":5363,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:48.461345+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:48.461345+0000","last_peered":"2024-04-26T02:27:48.461345+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:48.461345+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5285,"log_dups_size":0,"ondisk_log_size":5285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43274764,"num_objects":5283,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":42262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5294","reported_seq":5352,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.877507+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:46.877507+0000","last_peered":"2024-04-26T02:27:46.877507+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:46.877507+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43322130,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":42316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5277","reported_seq":5316,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.753597+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:46.753597+0000","last_peered":"2024-04-26T02:27:46.753597+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:46.753597+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43219206,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":42207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5355","reported_seq":5430,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:48.407881+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:48.407881+0000","last_peered":"2024-04-26T02:27:48.407881+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:48.407881+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43858182,"num_objects":5354,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":42831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3243}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5506","reported_seq":5559,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:46.674333+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:46.674333+0000","last_peered":"2024-04-26T02:27:46.674333+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:46.674333+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45068812,"num_objects":5502,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5504,"num_write_kb":44028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5400","reported_seq":7192,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:48.531674+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:48.531674+0000","last_peered":"2024-04-26T02:27:48.531674+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:48.531674+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44208652,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4200}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5391","reported_seq":5456,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:50.151149+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:50.151149+0000","last_peered":"2024-04-26T02:27:50.151149+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:50.151149+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44118540,"num_objects":5386,"num_object_clones":0,"num_object_copies":21544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":43100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":698694824,"num_objects":85296,"num_object_clones":0,"num_object_copies":341184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85323,"num_write_kb":682466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2967,"num_bytes_recovered":24068096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1274638336,"data_stored":1274638336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85350,"ondisk_log_size":85350,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631484,"kb_used_data":338804,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93740356,"statfs":{"total":96636764160,"available":95990124544,"internally_reserved":0,"allocated":346935296,"data_stored":346750174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":299691728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,6,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632512,"kb_used_data":340344,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93739328,"statfs":{"total":96636764160,"available":95989071872,"internally_reserved":0,"allocated":348512256,"data_stored":348327134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423312,"kb_used_data":220936,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93948528,"statfs":{"total":96636764160,"available":96203292672,"internally_reserved":0,"allocated":226238464,"data_stored":226053342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,8,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635440,"kb_used_data":342824,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93736400,"statfs":{"total":96636764160,"available":95986073600,"internally_reserved":0,"allocated":351051776,"data_stored":350866654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":351354880,"data_stored":351354880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226234368,"data_stored":226234368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349368320,"data_stored":349368320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347680768,"data_stored":347680768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:52.781 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 211 20 86395 86375 3.1966 2.03125 0.073314 0.0488694 2024-04-26T02:27:53.779 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 212 20 86665 86645 3.19147 2.10938 0.0714156 0.0489432 2024-04-26T02:27:54.779 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 213 20 86929 86909 3.18617 2.0625 0.0573736 0.0490218 2024-04-26T02:27:55.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:55.779 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 214 20 87165 87145 3.1799 1.84375 0.0377872 0.0491227 2024-04-26T02:27:56.779 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 215 20 87430 87410 3.17474 2.07031 0.145328 0.0492038 2024-04-26T02:27:57.134 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:57.134 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:57.271 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-26T02:27:56.525827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709229736,"num_objects":86582,"num_object_clones":0,"num_object_copies":346328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86609,"num_write_kb":692754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86636,"ondisk_log_size":86636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2325616,"kb_used_data":1265168,"kb_used_omap":14,"kb_used_meta":1060401,"kb_avail":375161744,"statfs":{"total":386547056640,"available":384165625856,"internally_reserved":0,"allocated":1295532032,"data_stored":1294791544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1085851187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,15,8,7,4,4,2,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24788992,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015289"},"pg_stats":[{"pgid":"1.f","version":"34'5276","reported_seq":5316,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.924083+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:51.924083+0000","last_peered":"2024-04-26T02:27:51.924083+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:51.924083+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43220992,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":42208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5509","reported_seq":5575,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.294577+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:55.294577+0000","last_peered":"2024-04-26T02:27:55.294577+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:55.294577+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45093388,"num_objects":5505,"num_object_clones":0,"num_object_copies":22020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5507,"num_write_kb":44052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5342","reported_seq":5420,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.407484+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:53.407484+0000","last_peered":"2024-04-26T02:27:53.407484+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:53.407484+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43735302,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5341,"num_write_kb":42718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3315}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5321","reported_seq":5384,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.468786+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:55.468786+0000","last_peered":"2024-04-26T02:27:55.468786+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:55.468786+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43553292,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5338","reported_seq":5394,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.892291+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:51.892291+0000","last_peered":"2024-04-26T02:27:51.892291+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:51.892291+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43700748,"num_objects":5335,"num_object_clones":0,"num_object_copies":21340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":42685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5319","reported_seq":5363,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.827489+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:51.827489+0000","last_peered":"2024-04-26T02:27:51.827489+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:51.827489+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43573248,"num_objects":5319,"num_object_clones":0,"num_object_copies":21276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5478","reported_seq":9720,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:55.285937+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:55.285937+0000","last_peered":"2024-04-26T02:27:55.285937+0000","last_clean":"2024-04-26T02:27:55.285937+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:55.285937+0000","last_undegraded":"2024-04-26T02:27:55.285937+0000","last_fullsized":"2024-04-26T02:27:55.285937+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:56.922678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823052,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5476,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5610","reported_seq":5676,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.390678+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:55.390678+0000","last_peered":"2024-04-26T02:27:55.390678+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:55.390678+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45912588,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5394","reported_seq":5451,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.743830+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:51.743830+0000","last_peered":"2024-04-26T02:27:51.743830+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:51.743830+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44114968,"num_objects":5386,"num_object_clones":0,"num_object_copies":21544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":43105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3382}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5372","reported_seq":5451,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.568322+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:53.568322+0000","last_peered":"2024-04-26T02:27:53.568322+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:53.568322+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5372,"log_dups_size":0,"ondisk_log_size":5372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43987468,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":42958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5365","reported_seq":5423,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.892755+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:51.892755+0000","last_peered":"2024-04-26T02:27:51.892755+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:51.892755+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43903762,"num_objects":5360,"num_object_clones":0,"num_object_copies":21440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":42884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5353","reported_seq":5392,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.853657+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:51.853657+0000","last_peered":"2024-04-26T02:27:51.853657+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:51.853657+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43841798,"num_objects":5352,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":42815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5434","reported_seq":5509,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.252016+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:53.252016+0000","last_peered":"2024-04-26T02:27:53.252016+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:53.252016+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505350,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":43463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5589","reported_seq":5642,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:51.798286+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:51.798286+0000","last_peered":"2024-04-26T02:27:51.798286+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:51.798286+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45748748,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5587,"num_write_kb":44692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3514}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5481","reported_seq":7419,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.539374+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:53.539374+0000","last_peered":"2024-04-26T02:27:53.539374+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:53.539374+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44872204,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5455","reported_seq":5520,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.426102+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:55.426102+0000","last_peered":"2024-04-26T02:27:55.426102+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:55.426102+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44642828,"num_objects":5450,"num_object_clones":0,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":709229736,"num_objects":86582,"num_object_clones":0,"num_object_copies":346328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86609,"num_write_kb":692754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1296371712,"data_stored":1296371712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86636,"ondisk_log_size":86636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628384,"kb_used_data":344024,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93743456,"statfs":{"total":96636764160,"available":95993298944,"internally_reserved":0,"allocated":352280576,"data_stored":352095454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":291172048},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,12,1,0,3,3,2,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630528,"kb_used_data":346680,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93741312,"statfs":{"total":96636764160,"available":95991103488,"internally_reserved":0,"allocated":355000320,"data_stored":354815198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434348,"kb_used_data":226404,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93937492,"statfs":{"total":96636764160,"available":96191991808,"internally_reserved":0,"allocated":231837696,"data_stored":231652574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212922063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632356,"kb_used_data":348060,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93739484,"statfs":{"total":96636764160,"available":95989231616,"internally_reserved":0,"allocated":356413440,"data_stored":356228318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356700160,"data_stored":356700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":231903232,"data_stored":231903232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354783232,"data_stored":354783232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352985088,"data_stored":352985088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:57.272 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:27:57.779 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 216 20 87699 87679 3.16977 2.10156 0.109667 0.0492773 2024-04-26T02:27:58.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 217 20 87962 87942 3.16457 2.05469 0.0334967 0.0493573 2024-04-26T02:27:59.146 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:27:59.146 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:27:59.171 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-04-26T02:27:58.526338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713882792,"num_objects":87150,"num_object_clones":0,"num_object_copies":348600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87177,"num_write_kb":697298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87204,"ondisk_log_size":87204,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2362812,"kb_used_data":1275996,"kb_used_omap":14,"kb_used_meta":1086769,"kb_avail":375124548,"statfs":{"total":386547056640,"available":384127537152,"internally_reserved":0,"allocated":1306619904,"data_stored":1305879416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1112852019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,11,14,8,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26042368,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":25432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011797"},"pg_stats":[{"pgid":"1.f","version":"34'5357","reported_seq":5399,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.954673+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:56.954673+0000","last_peered":"2024-04-26T02:27:56.954673+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:56.954673+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43884544,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5357,"num_write_kb":42856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5509","reported_seq":5575,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.294577+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:27:55.294577+0000","last_peered":"2024-04-26T02:27:55.294577+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:27:55.294577+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45093388,"num_objects":5505,"num_object_clones":0,"num_object_copies":22020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5507,"num_write_kb":44052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5342","reported_seq":5420,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.407484+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:53.407484+0000","last_peered":"2024-04-26T02:27:53.407484+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:53.407484+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43735302,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5341,"num_write_kb":42718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3315}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5321","reported_seq":5384,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.468786+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:27:55.468786+0000","last_peered":"2024-04-26T02:27:55.468786+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:27:55.468786+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43553292,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5418","reported_seq":5474,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.864918+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:56.864918+0000","last_peered":"2024-04-26T02:27:56.864918+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:56.864918+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356108,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":43325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5402","reported_seq":5446,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.533442+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:56.533442+0000","last_peered":"2024-04-26T02:27:56.533442+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:56.533442+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44253184,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5402,"num_write_kb":43216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5478","reported_seq":9720,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:27:55.285937+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:27:55.285937+0000","last_peered":"2024-04-26T02:27:55.285937+0000","last_clean":"2024-04-26T02:27:55.285937+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:27:55.285937+0000","last_undegraded":"2024-04-26T02:27:55.285937+0000","last_fullsized":"2024-04-26T02:27:55.285937+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:56.922678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823052,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5476,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5610","reported_seq":5676,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.390678+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:27:55.390678+0000","last_peered":"2024-04-26T02:27:55.390678+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:27:55.390678+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45912588,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5477","reported_seq":5534,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.962701+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:56.962701+0000","last_peered":"2024-04-26T02:27:56.962701+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:56.962701+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5477,"log_dups_size":0,"ondisk_log_size":5477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44794904,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5372","reported_seq":5451,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.568322+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:53.568322+0000","last_peered":"2024-04-26T02:27:53.568322+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:53.568322+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5372,"log_dups_size":0,"ondisk_log_size":5372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43987468,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":42958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5461","reported_seq":5519,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.954060+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:56.954060+0000","last_peered":"2024-04-26T02:27:56.954060+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:56.954060+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44690194,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5425","reported_seq":5464,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.639900+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:56.639900+0000","last_peered":"2024-04-26T02:27:56.639900+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:56.639900+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44431622,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5434","reported_seq":5509,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.252016+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:53.252016+0000","last_peered":"2024-04-26T02:27:53.252016+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:53.252016+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505350,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":43463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5662","reported_seq":5715,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.738449+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:56.738449+0000","last_peered":"2024-04-26T02:27:56.738449+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:56.738449+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46346764,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":45276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3587}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5481","reported_seq":7419,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:53.539374+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:53.539374+0000","last_peered":"2024-04-26T02:27:53.539374+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:53.539374+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44872204,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5455","reported_seq":5520,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:55.426102+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:27:55.426102+0000","last_peered":"2024-04-26T02:27:55.426102+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:27:55.426102+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44642828,"num_objects":5450,"num_object_clones":0,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":713882792,"num_objects":87150,"num_object_clones":0,"num_object_copies":348600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87177,"num_write_kb":697298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1307516928,"data_stored":1307516928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87204,"ondisk_log_size":87204,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637800,"kb_used_data":349280,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93734040,"statfs":{"total":96636764160,"available":95983656960,"internally_reserved":0,"allocated":357662720,"data_stored":357477598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":295431888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630528,"kb_used_data":346680,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93741312,"statfs":{"total":96636764160,"available":95991103488,"internally_reserved":0,"allocated":355000320,"data_stored":354815198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462128,"kb_used_data":231976,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93909712,"statfs":{"total":96636764160,"available":96163545088,"internally_reserved":0,"allocated":237543424,"data_stored":237358302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632356,"kb_used_data":348060,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93739484,"statfs":{"total":96636764160,"available":95989231616,"internally_reserved":0,"allocated":356413440,"data_stored":356228318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356700160,"data_stored":356700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237654016,"data_stored":237654016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354783232,"data_stored":354783232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358379520,"data_stored":358379520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5357", 'reported_seq': 5399, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.954673+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:27:56.954673+0000', 'last_peered': '2024-04-26T02:27:56.954673+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:27:56.954673+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5357, 'log_dups_size': 0, 'ondisk_log_size': 5357, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43884544, 'num_objects': 5357, 'num_object_clones': 0, 'num_object_copies': 21428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5357, 'num_write_kb': 42856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5509", 'reported_seq': 5575, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:55.294577+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:27:55.294577+0000', 'last_peered': '2024-04-26T02:27:55.294577+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:27:55.294577+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5509, 'log_dups_size': 0, 'ondisk_log_size': 5509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45093388, 'num_objects': 5505, 'num_object_clones': 0, 'num_object_copies': 22020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5507, 'num_write_kb': 44052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5342", 'reported_seq': 5420, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:53.407484+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:27:53.407484+0000', 'last_peered': '2024-04-26T02:27:53.407484+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:27:53.407484+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5342, 'log_dups_size': 0, 'ondisk_log_size': 5342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43735302, 'num_objects': 5339, 'num_object_clones': 0, 'num_object_copies': 21356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5341, 'num_write_kb': 42718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3315}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5321", 'reported_seq': 5384, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:55.468786+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:27:55.468786+0000', 'last_peered': '2024-04-26T02:27:55.468786+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:27:55.468786+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5321, 'log_dups_size': 0, 'ondisk_log_size': 5321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43553292, 'num_objects': 5317, 'num_object_clones': 0, 'num_object_copies': 21268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 42541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:59.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5418", 'reported_seq': 5474, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.864918+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:27:56.864918+0000', 'last_peered': '2024-04-26T02:27:56.864918+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:27:56.864918+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5418, 'log_dups_size': 0, 'ondisk_log_size': 5418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44356108, 'num_objects': 5415, 'num_object_clones': 0, 'num_object_copies': 21660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5416, 'num_write_kb': 43325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3404}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5402", 'reported_seq': 5446, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.533442+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:27:56.533442+0000', 'last_peered': '2024-04-26T02:27:56.533442+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:27:56.533442+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5402, 'log_dups_size': 0, 'ondisk_log_size': 5402, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44253184, 'num_objects': 5402, 'num_object_clones': 0, 'num_object_copies': 21608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5402, 'num_write_kb': 43216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5610", 'reported_seq': 5676, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:55.390678+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:27:55.390678+0000', 'last_peered': '2024-04-26T02:27:55.390678+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:27:55.390678+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5610, 'log_dups_size': 0, 'ondisk_log_size': 5610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45912588, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5608, 'num_write_kb': 44852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5477", 'reported_seq': 5534, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.962701+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:27:56.962701+0000', 'last_peered': '2024-04-26T02:27:56.962701+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:27:56.962701+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5477, 'log_dups_size': 0, 'ondisk_log_size': 5477, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44794904, 'num_objects': 5469, 'num_object_clones': 0, 'num_object_copies': 21876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5474, 'num_write_kb': 43769, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5372", 'reported_seq': 5451, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:53.568322+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:27:53.568322+0000', 'last_peered': '2024-04-26T02:27:53.568322+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:27:53.568322+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5372, 'log_dups_size': 0, 'ondisk_log_size': 5372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43987468, 'num_objects': 5370, 'num_object_clones': 0, 'num_object_copies': 21480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5370, 'num_write_kb': 42958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:27:59.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5461", 'reported_seq': 5519, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.954060+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:27:56.954060+0000', 'last_peered': '2024-04-26T02:27:56.954060+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:27:56.954060+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5461, 'log_dups_size': 0, 'ondisk_log_size': 5461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44690194, 'num_objects': 5456, 'num_object_clones': 0, 'num_object_copies': 21824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5458, 'num_write_kb': 43652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3439}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5425", 'reported_seq': 5464, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.639900+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:27:56.639900+0000', 'last_peered': '2024-04-26T02:27:56.639900+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:27:56.639900+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5425, 'log_dups_size': 0, 'ondisk_log_size': 5425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44431622, 'num_objects': 5424, 'num_object_clones': 0, 'num_object_copies': 21696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5424, 'num_write_kb': 43391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5434", 'reported_seq': 5509, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:53.252016+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:27:53.252016+0000', 'last_peered': '2024-04-26T02:27:53.252016+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:27:53.252016+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5434, 'log_dups_size': 0, 'ondisk_log_size': 5434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44505350, 'num_objects': 5433, 'num_object_clones': 0, 'num_object_copies': 21732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5433, 'num_write_kb': 43463, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5662", 'reported_seq': 5715, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:56.738449+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:27:56.738449+0000', 'last_peered': '2024-04-26T02:27:56.738449+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:27:56.738449+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5662, 'log_dups_size': 0, 'ondisk_log_size': 5662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46346764, 'num_objects': 5658, 'num_object_clones': 0, 'num_object_copies': 22632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5660, 'num_write_kb': 45276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5481", 'reported_seq': 7419, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:53.539374+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:27:53.539374+0000', 'last_peered': '2024-04-26T02:27:53.539374+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:27:53.539374+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5481, 'log_dups_size': 0, 'ondisk_log_size': 5481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44872204, 'num_objects': 5478, 'num_object_clones': 0, 'num_object_copies': 21912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1124, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5479, 'num_write_kb': 43829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 931, 'num_bytes_recovered': 7610368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:27:59.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5455", 'reported_seq': 5520, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:27:55.426102+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:27:55.426102+0000', 'last_peered': '2024-04-26T02:27:55.426102+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:27:55.426102+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5455, 'log_dups_size': 0, 'ondisk_log_size': 5455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44642828, 'num_objects': 5450, 'num_object_clones': 0, 'num_object_copies': 21800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5453, 'num_write_kb': 43612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:27:59.175 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:27:59.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 218 20 88197 88177 3.15847 1.83594 0.0685342 0.0494527 2024-04-26T02:28:00.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:00.570 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29727/348600 objects degraded (8.528%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":277,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87150,"data_bytes":713882792,"bytes_used":2419519488,"bytes_avail":384127537152,"bytes_total":386547056640,"degraded_objects":29727,"degraded_total":348600,"degraded_ratio":0.085275387263339067,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":113893,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"write_bytes_sec":2168065,"read_op_per_sec":0,"write_op_per_sec":264},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:00.695 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:00.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:00.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 219 20 88450 88430 3.15308 1.97656 0.0593061 0.0495404 2024-04-26T02:28:01.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:2024-04-26T02:28:01.784852+0000 min lat: 0.0028054 max lat: 2.17928 avg lat: 0.0496275 2024-04-26T02:28:01.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-26T02:28:01.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 220 20 88697 88677 3.14752 1.92969 0.0704719 0.0496275 2024-04-26T02:28:02.103 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:02.104 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:02.166 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-04-26T02:28:00.526868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":719887528,"num_objects":87883,"num_object_clones":0,"num_object_copies":351532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87910,"num_write_kb":703162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3127,"num_bytes_recovered":25378816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87937,"ondisk_log_size":87937,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2399572,"kb_used_data":1286388,"kb_used_omap":14,"kb_used_meta":1113137,"kb_avail":375087788,"statfs":{"total":386547056640,"available":384089894912,"internally_reserved":0,"allocated":1317261312,"data_stored":1316520824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1139852851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,13,16,8,11,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27353088,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":26712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009970"},"pg_stats":[{"pgid":"1.f","version":"34'5357","reported_seq":5399,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.954673+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:27:56.954673+0000","last_peered":"2024-04-26T02:27:56.954673+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:27:56.954673+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43884544,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5357,"num_write_kb":42856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5579","reported_seq":5645,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:00.303924+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:00.303924+0000","last_peered":"2024-04-26T02:28:00.303924+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:00.303924+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45666828,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":44612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5432","reported_seq":5510,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:58.462885+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:27:58.462885+0000","last_peered":"2024-04-26T02:27:58.462885+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:27:58.462885+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44472582,"num_objects":5429,"num_object_clones":0,"num_object_copies":21716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":43438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3405}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5397","reported_seq":5460,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:00.394355+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:00.394355+0000","last_peered":"2024-04-26T02:28:00.394355+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:00.394355+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44175884,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":43149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3425}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5418","reported_seq":5474,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.864918+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:27:56.864918+0000","last_peered":"2024-04-26T02:27:56.864918+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:27:56.864918+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356108,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":43325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5402","reported_seq":5446,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.533442+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:27:56.533442+0000","last_peered":"2024-04-26T02:27:56.533442+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:27:56.533442+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44253184,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5402,"num_write_kb":43216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5550","reported_seq":9792,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:00.321647+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:00.321647+0000","last_peered":"2024-04-26T02:28:00.321647+0000","last_clean":"2024-04-26T02:28:00.321647+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:00.321647+0000","last_undegraded":"2024-04-26T02:28:00.321647+0000","last_fullsized":"2024-04-26T02:28:00.321647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:01.999500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45412876,"num_objects":5544,"num_object_clones":0,"num_object_copies":22176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5548,"num_write_kb":44364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5693","reported_seq":5759,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:00.384498+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:00.384498+0000","last_peered":"2024-04-26T02:28:00.384498+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:00.384498+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46592524,"num_objects":5688,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5691,"num_write_kb":45516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3547}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5477","reported_seq":5534,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.962701+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:27:56.962701+0000","last_peered":"2024-04-26T02:27:56.962701+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:27:56.962701+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5477,"log_dups_size":0,"ondisk_log_size":5477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44794904,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5456","reported_seq":5535,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:58.552806+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:27:58.552806+0000","last_peered":"2024-04-26T02:27:58.552806+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:27:58.552806+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44675596,"num_objects":5454,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5454,"num_write_kb":43630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5461","reported_seq":5519,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.954060+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:27:56.954060+0000","last_peered":"2024-04-26T02:27:56.954060+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:27:56.954060+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44690194,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5425","reported_seq":5464,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.639900+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:27:56.639900+0000","last_peered":"2024-04-26T02:27:56.639900+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:27:56.639900+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44431622,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5509","reported_seq":5584,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:58.402849+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:27:58.402849+0000","last_peered":"2024-04-26T02:27:58.402849+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:27:58.402849+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45119750,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5508,"num_write_kb":44063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5662","reported_seq":5715,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:56.738449+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:27:56.738449+0000","last_peered":"2024-04-26T02:27:56.738449+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:27:56.738449+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46346764,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":45276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3587}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5565","reported_seq":7676,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:27:58.553098+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:27:58.553098+0000","last_peered":"2024-04-26T02:27:58.553098+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:27:58.553098+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45560332,"num_objects":5562,"num_object_clones":0,"num_object_copies":22248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":44501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1018,"num_bytes_recovered":8323072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4525}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5554","reported_seq":5619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:00.336040+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:00.336040+0000","last_peered":"2024-04-26T02:28:00.336040+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:00.336040+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5554,"log_dups_size":0,"ondisk_log_size":5554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45453836,"num_objects":5549,"num_object_clones":0,"num_object_copies":22196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":44404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":719887528,"num_objects":87883,"num_object_clones":0,"num_object_copies":351532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87910,"num_write_kb":703162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3127,"num_bytes_recovered":25378816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1318125568,"data_stored":1318125568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87937,"ondisk_log_size":87937,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637800,"kb_used_data":349280,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93734040,"statfs":{"total":96636764160,"available":95983656960,"internally_reserved":0,"allocated":357662720,"data_stored":357477598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":295431888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,4,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639852,"kb_used_data":351844,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93731988,"statfs":{"total":96636764160,"available":95981555712,"internally_reserved":0,"allocated":360288256,"data_stored":360103134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,5,1,5,0,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462128,"kb_used_data":231976,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93909712,"statfs":{"total":96636764160,"available":96163545088,"internally_reserved":0,"allocated":237543424,"data_stored":237358302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659792,"kb_used_data":353288,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93712048,"statfs":{"total":96636764160,"available":95961137152,"internally_reserved":0,"allocated":361766912,"data_stored":361581790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361975808,"data_stored":361975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237654016,"data_stored":237654016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360116224,"data_stored":360116224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358379520,"data_stored":358379520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:02.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 221 20 88964 88944 3.14272 2.08594 0.0761023 0.0497037 2024-04-26T02:28:03.784 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 222 20 89238 89218 3.1382 2.14062 0.0340515 0.0497731 2024-04-26T02:28:04.789 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 223 20 89516 89496 3.13387 2.17188 0.1908 0.0498428 2024-04-26T02:28:05.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:05.785 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 224 20 89773 89753 3.12884 2.00781 0.0495326 0.0499254 2024-04-26T02:28:06.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 225 20 90029 90009 3.12381 2 0.0440692 0.0500031 2024-04-26T02:28:06.991 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:06.992 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:07.097 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-26T02:28:06.528682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":730619048,"num_objects":89193,"num_object_clones":0,"num_object_copies":356772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89220,"num_write_kb":713642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3190,"num_bytes_recovered":25894912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89247,"ondisk_log_size":89247,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567616,"kb_used_data":1307292,"kb_used_omap":14,"kb_used_meta":1260273,"kb_avail":374919744,"statfs":{"total":386547056640,"available":383917817856,"internally_reserved":0,"allocated":1338667008,"data_stored":1337926520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1290520115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,0,8,12,6,16,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24715264,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"1.f","version":"34'5435","reported_seq":5477,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.015287+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:02.015287+0000","last_peered":"2024-04-26T02:28:02.015287+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:02.015287+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44523520,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":43480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5675","reported_seq":5741,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471833+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:05.471833+0000","last_peered":"2024-04-26T02:28:05.471833+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:05.471833+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46453260,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5514","reported_seq":5592,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.605974+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:03.605974+0000","last_peered":"2024-04-26T02:28:03.605974+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:03.605974+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45144326,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5494","reported_seq":5557,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.209015+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:05.209015+0000","last_peered":"2024-04-26T02:28:05.209015+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:05.209015+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44970508,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5497","reported_seq":5553,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.953481+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:01.953481+0000","last_peered":"2024-04-26T02:28:01.953481+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:01.953481+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003276,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":43957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5479","reported_seq":5523,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.934834+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:01.934834+0000","last_peered":"2024-04-26T02:28:01.934834+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:01.934834+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44883968,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5640","reported_seq":9882,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:05.535017+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:05.535017+0000","last_peered":"2024-04-26T02:28:05.535017+0000","last_clean":"2024-04-26T02:28:05.535017+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:05.535017+0000","last_undegraded":"2024-04-26T02:28:05.535017+0000","last_fullsized":"2024-04-26T02:28:05.535017+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:07.048776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46150156,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":45084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5766","reported_seq":5832,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471011+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:05.471011+0000","last_peered":"2024-04-26T02:28:05.471011+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:05.471011+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47190540,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5554","reported_seq":5611,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.979884+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:01.979884+0000","last_peered":"2024-04-26T02:28:01.979884+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:01.979884+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5554,"log_dups_size":0,"ondisk_log_size":5554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45425688,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5539","reported_seq":5619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.653545+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:03.653545+0000","last_peered":"2024-04-26T02:28:03.653545+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:03.653545+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45355532,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":44294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5543","reported_seq":5601,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.935157+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:01.935157+0000","last_peered":"2024-04-26T02:28:01.935157+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:01.935157+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45361938,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":44308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5512","reported_seq":5551,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.022706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:02.022706+0000","last_peered":"2024-04-26T02:28:02.022706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:02.022706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45144326,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5511,"num_write_kb":44087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3523}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5601","reported_seq":5676,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.570987+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:03.570987+0000","last_peered":"2024-04-26T02:28:03.570987+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:03.570987+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45873414,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5737","reported_seq":5790,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.059212+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:02.059212+0000","last_peered":"2024-04-26T02:28:02.059212+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:02.059212+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961164,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5735,"num_write_kb":45876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5644","reported_seq":7882,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.661278+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:03.661278+0000","last_peered":"2024-04-26T02:28:03.661278+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:03.661278+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46207500,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5642,"num_write_kb":45133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1081,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5617","reported_seq":5682,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.343165+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:05.343165+0000","last_peered":"2024-04-26T02:28:05.343165+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:05.343165+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45969932,"num_objects":5612,"num_object_clones":0,"num_object_copies":22448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":730619048,"num_objects":89193,"num_object_clones":0,"num_object_copies":356772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89220,"num_write_kb":713642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3190,"num_bytes_recovered":25894912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339641856,"data_stored":1339641856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89247,"ondisk_log_size":89247,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683276,"kb_used_data":354496,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93688564,"statfs":{"total":96636764160,"available":95937089536,"internally_reserved":0,"allocated":363003904,"data_stored":362818782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,4,4,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703240,"kb_used_data":356928,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93668600,"statfs":{"total":96636764160,"available":95916646400,"internally_reserved":0,"allocated":365494272,"data_stored":365309150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1,8,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475744,"kb_used_data":237272,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93896096,"statfs":{"total":96636764160,"available":96149602304,"internally_reserved":0,"allocated":242966528,"data_stored":242781406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":358596,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":367202304,"data_stored":367017182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367419392,"data_stored":367419392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243146752,"data_stored":243146752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365404160,"data_stored":365404160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363671552,"data_stored":363671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:07.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:07.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 226 20 90284 90264 3.1188 1.99219 0.110844 0.0500803 2024-04-26T02:28:08.456 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:08.457 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:08.574 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-26T02:28:06.528682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":730619048,"num_objects":89193,"num_object_clones":0,"num_object_copies":356772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89220,"num_write_kb":713642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3190,"num_bytes_recovered":25894912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89247,"ondisk_log_size":89247,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567616,"kb_used_data":1307292,"kb_used_omap":14,"kb_used_meta":1260273,"kb_avail":374919744,"statfs":{"total":386547056640,"available":383917817856,"internally_reserved":0,"allocated":1338667008,"data_stored":1337926520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1290520115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,0,8,12,6,16,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24715264,"num_objects":3017,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":24136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"1.f","version":"34'5435","reported_seq":5477,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.015287+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:02.015287+0000","last_peered":"2024-04-26T02:28:02.015287+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:02.015287+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44523520,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":43480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5675","reported_seq":5741,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471833+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:05.471833+0000","last_peered":"2024-04-26T02:28:05.471833+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:05.471833+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46453260,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5514","reported_seq":5592,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.605974+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:03.605974+0000","last_peered":"2024-04-26T02:28:03.605974+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:03.605974+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45144326,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5494","reported_seq":5557,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.209015+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:05.209015+0000","last_peered":"2024-04-26T02:28:05.209015+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:05.209015+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44970508,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5497","reported_seq":5553,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.953481+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:01.953481+0000","last_peered":"2024-04-26T02:28:01.953481+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:01.953481+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45003276,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":43957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5479","reported_seq":5523,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.934834+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:01.934834+0000","last_peered":"2024-04-26T02:28:01.934834+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:01.934834+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44883968,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":43832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5640","reported_seq":9882,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:05.535017+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:05.535017+0000","last_peered":"2024-04-26T02:28:05.535017+0000","last_clean":"2024-04-26T02:28:05.535017+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:05.535017+0000","last_undegraded":"2024-04-26T02:28:05.535017+0000","last_fullsized":"2024-04-26T02:28:05.535017+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:07.048776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46150156,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":45084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5766","reported_seq":5832,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471011+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:05.471011+0000","last_peered":"2024-04-26T02:28:05.471011+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:05.471011+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47190540,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5554","reported_seq":5611,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.979884+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:01.979884+0000","last_peered":"2024-04-26T02:28:01.979884+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:01.979884+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5554,"log_dups_size":0,"ondisk_log_size":5554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45425688,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5539","reported_seq":5619,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.653545+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:03.653545+0000","last_peered":"2024-04-26T02:28:03.653545+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:03.653545+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45355532,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":44294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3500}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5543","reported_seq":5601,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:01.935157+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:01.935157+0000","last_peered":"2024-04-26T02:28:01.935157+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:01.935157+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45361938,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":44308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5512","reported_seq":5551,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.022706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:02.022706+0000","last_peered":"2024-04-26T02:28:02.022706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:02.022706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45144326,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5511,"num_write_kb":44087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3523}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5601","reported_seq":5676,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.570987+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:03.570987+0000","last_peered":"2024-04-26T02:28:03.570987+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:03.570987+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45873414,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5737","reported_seq":5790,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:02.059212+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:02.059212+0000","last_peered":"2024-04-26T02:28:02.059212+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:02.059212+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46961164,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5735,"num_write_kb":45876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5644","reported_seq":7882,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:03.661278+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:03.661278+0000","last_peered":"2024-04-26T02:28:03.661278+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:03.661278+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46207500,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5642,"num_write_kb":45133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1081,"num_bytes_recovered":8839168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5617","reported_seq":5682,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.343165+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:05.343165+0000","last_peered":"2024-04-26T02:28:05.343165+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:05.343165+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45969932,"num_objects":5612,"num_object_clones":0,"num_object_copies":22448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":730619048,"num_objects":89193,"num_object_clones":0,"num_object_copies":356772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89220,"num_write_kb":713642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3190,"num_bytes_recovered":25894912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1339641856,"data_stored":1339641856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89247,"ondisk_log_size":89247,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683276,"kb_used_data":354496,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93688564,"statfs":{"total":96636764160,"available":95937089536,"internally_reserved":0,"allocated":363003904,"data_stored":362818782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,4,4,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703240,"kb_used_data":356928,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93668600,"statfs":{"total":96636764160,"available":95916646400,"internally_reserved":0,"allocated":365494272,"data_stored":365309150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1,8,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475744,"kb_used_data":237272,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93896096,"statfs":{"total":96636764160,"available":96149602304,"internally_reserved":0,"allocated":242966528,"data_stored":242781406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":358596,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":367202304,"data_stored":367017182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367419392,"data_stored":367419392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243146752,"data_stored":243146752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365404160,"data_stored":365404160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363671552,"data_stored":363671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5435", 'reported_seq': 5477, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:02.015287+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:02.015287+0000', 'last_peered': '2024-04-26T02:28:02.015287+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:02.015287+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5435, 'log_dups_size': 0, 'ondisk_log_size': 5435, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44523520, 'num_objects': 5435, 'num_object_clones': 0, 'num_object_copies': 21740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5435, 'num_write_kb': 43480, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3418}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5675", 'reported_seq': 5741, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:05.471833+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:05.471833+0000', 'last_peered': '2024-04-26T02:28:05.471833+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:05.471833+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5675, 'log_dups_size': 0, 'ondisk_log_size': 5675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46453260, 'num_objects': 5671, 'num_object_clones': 0, 'num_object_copies': 22684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5673, 'num_write_kb': 45380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:08.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5514", 'reported_seq': 5592, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:03.605974+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:03.605974+0000', 'last_peered': '2024-04-26T02:28:03.605974+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:03.605974+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5514, 'log_dups_size': 0, 'ondisk_log_size': 5514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45144326, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5513, 'num_write_kb': 44094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3487}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5494", 'reported_seq': 5557, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:05.209015+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:05.209015+0000', 'last_peered': '2024-04-26T02:28:05.209015+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:05.209015+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5494, 'log_dups_size': 0, 'ondisk_log_size': 5494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44970508, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 21960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5492, 'num_write_kb': 43925, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5497", 'reported_seq': 5553, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:01.953481+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:01.953481+0000', 'last_peered': '2024-04-26T02:28:01.953481+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:01.953481+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5497, 'log_dups_size': 0, 'ondisk_log_size': 5497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45003276, 'num_objects': 5494, 'num_object_clones': 0, 'num_object_copies': 21976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5495, 'num_write_kb': 43957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5479", 'reported_seq': 5523, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:01.934834+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:01.934834+0000', 'last_peered': '2024-04-26T02:28:01.934834+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:01.934834+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5479, 'log_dups_size': 0, 'ondisk_log_size': 5479, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44883968, 'num_objects': 5479, 'num_object_clones': 0, 'num_object_copies': 21916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5479, 'num_write_kb': 43832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3428}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5766", 'reported_seq': 5832, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:05.471011+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:05.471011+0000', 'last_peered': '2024-04-26T02:28:05.471011+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:05.471011+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5766, 'log_dups_size': 0, 'ondisk_log_size': 5766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47190540, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5764, 'num_write_kb': 46100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5554", 'reported_seq': 5611, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:01.979884+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:01.979884+0000', 'last_peered': '2024-04-26T02:28:01.979884+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:01.979884+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5554, 'log_dups_size': 0, 'ondisk_log_size': 5554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45425688, 'num_objects': 5546, 'num_object_clones': 0, 'num_object_copies': 22184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5551, 'num_write_kb': 44385, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3542}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5539", 'reported_seq': 5619, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:03.653545+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:03.653545+0000', 'last_peered': '2024-04-26T02:28:03.653545+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:03.653545+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5539, 'log_dups_size': 0, 'ondisk_log_size': 5539, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45355532, 'num_objects': 5537, 'num_object_clones': 0, 'num_object_copies': 22148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5537, 'num_write_kb': 44294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3500}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5543", 'reported_seq': 5601, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:01.935157+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:01.935157+0000', 'last_peered': '2024-04-26T02:28:01.935157+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:01.935157+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5543, 'log_dups_size': 0, 'ondisk_log_size': 5543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45361938, 'num_objects': 5538, 'num_object_clones': 0, 'num_object_copies': 22152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5540, 'num_write_kb': 44308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3521}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5512", 'reported_seq': 5551, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:02.022706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:02.022706+0000', 'last_peered': '2024-04-26T02:28:02.022706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:02.022706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5512, 'log_dups_size': 0, 'ondisk_log_size': 5512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45144326, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5511, 'num_write_kb': 44087, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5601", 'reported_seq': 5676, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:03.570987+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:03.570987+0000', 'last_peered': '2024-04-26T02:28:03.570987+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:03.570987+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5601, 'log_dups_size': 0, 'ondisk_log_size': 5601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45873414, 'num_objects': 5600, 'num_object_clones': 0, 'num_object_copies': 22400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5600, 'num_write_kb': 44799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5737", 'reported_seq': 5790, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:02.059212+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:02.059212+0000', 'last_peered': '2024-04-26T02:28:02.059212+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:02.059212+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5737, 'log_dups_size': 0, 'ondisk_log_size': 5737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46961164, 'num_objects': 5733, 'num_object_clones': 0, 'num_object_copies': 22932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5735, 'num_write_kb': 45876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:08.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5644", 'reported_seq': 7882, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:03.661278+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:03.661278+0000', 'last_peered': '2024-04-26T02:28:03.661278+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:03.661278+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5644, 'log_dups_size': 0, 'ondisk_log_size': 5644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46207500, 'num_objects': 5641, 'num_object_clones': 0, 'num_object_copies': 22564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 974, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5642, 'num_write_kb': 45133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1081, 'num_bytes_recovered': 8839168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:08.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:08.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5617", 'reported_seq': 5682, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:05.343165+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:05.343165+0000', 'last_peered': '2024-04-26T02:28:05.343165+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:05.343165+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5617, 'log_dups_size': 0, 'ondisk_log_size': 5617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45969932, 'num_objects': 5612, 'num_object_clones': 0, 'num_object_copies': 22448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5615, 'num_write_kb': 44908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3548}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:08.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:08.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 227 20 90539 90519 3.11384 1.99219 0.0946928 0.0501686 2024-04-26T02:28:09.786 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 228 20 90786 90766 3.10865 1.92969 0.0866615 0.0502455 2024-04-26T02:28:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:10.253 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29577/359184 objects degraded (8.234%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89796,"data_bytes":735558824,"bytes_used":2633502720,"bytes_avail":383913553920,"bytes_total":386547056640,"degraded_objects":29577,"degraded_total":359184,"degraded_ratio":0.082344981959107291,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":102371,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"write_bytes_sec":2193474,"read_op_per_sec":0,"write_op_per_sec":267},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:10.324 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:10.324 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:10.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 229 20 91045 91025 3.10391 2.02344 0.0338276 0.0503222 2024-04-26T02:28:11.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 230 20 91301 91281 3.09911 2 0.0524103 0.0504041 2024-04-26T02:28:11.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:11.949 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:12.051 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-04-26T02:28:10.529701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738081960,"num_objects":90104,"num_object_clones":0,"num_object_copies":360416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90131,"num_write_kb":720930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90158,"ondisk_log_size":90158,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564452,"kb_used_data":1323524,"kb_used_omap":14,"kb_used_meta":1240881,"kb_avail":374922908,"statfs":{"total":386547056640,"available":383921057792,"internally_reserved":0,"allocated":1355288576,"data_stored":1354548088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1270662707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,10,12,16,14,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24199168,"num_objects":2954,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003363"},"pg_stats":[{"pgid":"1.f","version":"34'5528","reported_seq":5567,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.123437+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:07.123437+0000","last_peered":"2024-04-26T02:28:07.123437+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:07.123437+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5528,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5675","reported_seq":5741,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471833+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:05.471833+0000","last_peered":"2024-04-26T02:28:05.471833+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:05.471833+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46453260,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":45380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5597","reported_seq":5675,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:08.658214+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:08.658214+0000","last_peered":"2024-04-26T02:28:08.658214+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:08.658214+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45824262,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3570}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5494","reported_seq":5557,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.209015+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:05.209015+0000","last_peered":"2024-04-26T02:28:05.209015+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:05.209015+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44970508,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5591","reported_seq":5647,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.024212+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:07.024212+0000","last_peered":"2024-04-26T02:28:07.024212+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:07.024212+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45773324,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5589,"num_write_kb":44709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5559","reported_seq":5603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:06.952859+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:06.952859+0000","last_peered":"2024-04-26T02:28:06.952859+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:06.952859+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45539328,"num_objects":5559,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5640","reported_seq":9882,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:05.535017+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:05.535017+0000","last_peered":"2024-04-26T02:28:05.535017+0000","last_clean":"2024-04-26T02:28:05.535017+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:05.535017+0000","last_undegraded":"2024-04-26T02:28:05.535017+0000","last_fullsized":"2024-04-26T02:28:05.535017+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:07.048776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46150156,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":45084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5766","reported_seq":5832,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.471011+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:05.471011+0000","last_peered":"2024-04-26T02:28:05.471011+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:05.471011+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47190540,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5630","reported_seq":5687,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.001642+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:07.001642+0000","last_peered":"2024-04-26T02:28:07.001642+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:07.001642+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46048280,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":44993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5601","reported_seq":5681,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:08.685470+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:08.685470+0000","last_peered":"2024-04-26T02:28:08.685470+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:08.685470+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45863436,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":44790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5639","reported_seq":5697,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.123783+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:07.123783+0000","last_peered":"2024-04-26T02:28:07.123783+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:07.123783+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46148370,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":45076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5583","reported_seq":5621,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.122972+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:07.122972+0000","last_peered":"2024-04-26T02:28:07.122972+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:07.122972+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45725958,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":44655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3594}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5682","reported_seq":5757,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:08.480855+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:08.480855+0000","last_peered":"2024-04-26T02:28:08.480855+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:08.480855+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46536966,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3570}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5830","reported_seq":5883,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:07.093323+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:07.093323+0000","last_peered":"2024-04-26T02:28:07.093323+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:07.093323+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723020,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3755}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5726","reported_seq":8106,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:08.715348+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:08.715348+0000","last_peered":"2024-04-26T02:28:08.715348+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:08.715348+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46879244,"num_objects":5723,"num_object_clones":0,"num_object_copies":22892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":45789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1152,"num_bytes_recovered":9420800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5617","reported_seq":5682,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:05.343165+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:05.343165+0000","last_peered":"2024-04-26T02:28:05.343165+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:05.343165+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45969932,"num_objects":5612,"num_object_clones":0,"num_object_copies":22448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738081960,"num_objects":90104,"num_object_clones":0,"num_object_copies":360416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90131,"num_write_kb":720930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":26476544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356247040,"data_stored":1356247040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90158,"ondisk_log_size":90158,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":359840,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":368476160,"data_stored":368291038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,6,2,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695912,"kb_used_data":362080,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93675928,"statfs":{"total":96636764160,"available":95924150272,"internally_reserved":0,"allocated":370769920,"data_stored":370584798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469000,"kb_used_data":243008,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93902840,"statfs":{"total":96636764160,"available":96156508160,"internally_reserved":0,"allocated":248840192,"data_stored":248655070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":358596,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":367202304,"data_stored":367017182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,6,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367419392,"data_stored":367419392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248893440,"data_stored":248893440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370761728,"data_stored":370761728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369172480,"data_stored":369172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:12.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 231 20 91563 91543 3.09456 2.04688 0.0783399 0.0504732 2024-04-26T02:28:13.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 232 20 91830 91810 3.09021 2.08594 0.06759 0.0505512 2024-04-26T02:28:14.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 233 20 92080 92060 3.08533 1.95312 0.0516044 0.0506174 2024-04-26T02:28:15.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:15.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 234 20 92332 92312 3.08056 1.96875 0.26246 0.0506954 2024-04-26T02:28:16.610 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:16.611 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:16.709 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-04-26T02:28:16.531279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751934632,"num_objects":91795,"num_object_clones":0,"num_object_copies":367180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91822,"num_write_kb":734458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3337,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91849,"ondisk_log_size":91849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607904,"kb_used_data":1350336,"kb_used_omap":14,"kb_used_meta":1257521,"kb_avail":374879456,"statfs":{"total":386547056640,"available":383876562944,"internally_reserved":0,"allocated":1382744064,"data_stored":1382003576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1287702067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,1,2,12,11,7,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24748032,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3021,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.f","version":"34'5609","reported_seq":5650,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.265922+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:12.265922+0000","last_peered":"2024-04-26T02:28:12.265922+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:12.265922+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45948928,"num_objects":5609,"num_object_clones":0,"num_object_copies":22436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5609,"num_write_kb":44872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5820","reported_seq":5886,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.506878+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:15.506878+0000","last_peered":"2024-04-26T02:28:15.506878+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:15.506878+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47641100,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5683","reported_seq":5762,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.753114+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:13.753114+0000","last_peered":"2024-04-26T02:28:13.753114+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:13.753114+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46528774,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5654","reported_seq":5716,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.626525+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:15.626525+0000","last_peered":"2024-04-26T02:28:15.626525+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:15.626525+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281228,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5668","reported_seq":5724,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.219697+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:12.219697+0000","last_peered":"2024-04-26T02:28:12.219697+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:12.219697+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46404108,"num_objects":5665,"num_object_clones":0,"num_object_copies":22660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5650","reported_seq":5694,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.184296+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:12.184296+0000","last_peered":"2024-04-26T02:28:12.184296+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:12.184296+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46284800,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":45200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5789","reported_seq":10031,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:15.431618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:15.431618+0000","last_peered":"2024-04-26T02:28:15.431618+0000","last_clean":"2024-04-26T02:28:15.431618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:15.431618+0000","last_undegraded":"2024-04-26T02:28:15.431618+0000","last_fullsized":"2024-04-26T02:28:15.431618+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.963474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47370764,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5925","reported_seq":5991,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.576484+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:15.576484+0000","last_peered":"2024-04-26T02:28:15.576484+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:15.576484+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48493068,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":47372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5717","reported_seq":5774,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.181439+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:12.181439+0000","last_peered":"2024-04-26T02:28:12.181439+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:12.181439+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46760984,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5714,"num_write_kb":45689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3705}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5676","reported_seq":5756,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.615137+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:13.615137+0000","last_peered":"2024-04-26T02:28:13.615137+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:13.615137+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46477836,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5716","reported_seq":5774,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.277313+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:12.277313+0000","last_peered":"2024-04-26T02:28:12.277313+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:12.277313+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46779154,"num_objects":5711,"num_object_clones":0,"num_object_copies":22844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5675","reported_seq":5712,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.221796+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:12.221796+0000","last_peered":"2024-04-26T02:28:12.221796+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:12.221796+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46479622,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5756","reported_seq":5831,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.571316+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:13.571316+0000","last_peered":"2024-04-26T02:28:13.571316+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:13.571316+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143174,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":46039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5920","reported_seq":5973,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.115576+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:12.115576+0000","last_peered":"2024-04-26T02:28:12.115576+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:12.115576+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48460300,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3845}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5808","reported_seq":8340,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.747357+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:13.747357+0000","last_peered":"2024-04-26T02:28:13.747357+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:13.747357+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47550988,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1228,"num_bytes_recovered":10043392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4978}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5783","reported_seq":5848,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.539814+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:15.539814+0000","last_peered":"2024-04-26T02:28:15.539814+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:15.539814+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329804,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":46236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751934632,"num_objects":91795,"num_object_clones":0,"num_object_copies":367180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91822,"num_write_kb":734458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3337,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1383067648,"data_stored":1383067648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91849,"ondisk_log_size":91849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708544,"kb_used_data":365880,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93663296,"statfs":{"total":96636764160,"available":95911215104,"internally_reserved":0,"allocated":374661120,"data_stored":374475998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,7,9,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709436,"kb_used_data":367284,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93662404,"statfs":{"total":96636764160,"available":95910301696,"internally_reserved":0,"allocated":376098816,"data_stored":375913694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478500,"kb_used_data":248348,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93893340,"statfs":{"total":96636764160,"available":96146780160,"internally_reserved":0,"allocated":254308352,"data_stored":254123230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711424,"kb_used_data":368824,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93660416,"statfs":{"total":96636764160,"available":95908265984,"internally_reserved":0,"allocated":377675776,"data_stored":377490654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378019840,"data_stored":378019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254423040,"data_stored":254423040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376115200,"data_stored":376115200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374509568,"data_stored":374509568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:16.710 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:16.787 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 235 20 92585 92565 3.07586 1.97656 0.0122904 0.050773 2024-04-26T02:28:17.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 236 20 92821 92801 3.07061 1.84375 0.066118 0.0508686 2024-04-26T02:28:17.854 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:17.854 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:18.004 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-04-26T02:28:16.531279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751934632,"num_objects":91795,"num_object_clones":0,"num_object_copies":367180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91822,"num_write_kb":734458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3337,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91849,"ondisk_log_size":91849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607904,"kb_used_data":1350336,"kb_used_omap":14,"kb_used_meta":1257521,"kb_avail":374879456,"statfs":{"total":386547056640,"available":383876562944,"internally_reserved":0,"allocated":1382744064,"data_stored":1382003576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1287702067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,1,2,12,11,7,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24748032,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3021,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.f","version":"34'5609","reported_seq":5650,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.265922+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:12.265922+0000","last_peered":"2024-04-26T02:28:12.265922+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:12.265922+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45948928,"num_objects":5609,"num_object_clones":0,"num_object_copies":22436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5609,"num_write_kb":44872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5820","reported_seq":5886,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.506878+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:15.506878+0000","last_peered":"2024-04-26T02:28:15.506878+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:15.506878+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47641100,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5683","reported_seq":5762,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.753114+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:13.753114+0000","last_peered":"2024-04-26T02:28:13.753114+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:13.753114+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46528774,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5654","reported_seq":5716,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.626525+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:15.626525+0000","last_peered":"2024-04-26T02:28:15.626525+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:15.626525+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281228,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5668","reported_seq":5724,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.219697+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:12.219697+0000","last_peered":"2024-04-26T02:28:12.219697+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:12.219697+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46404108,"num_objects":5665,"num_object_clones":0,"num_object_copies":22660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5650","reported_seq":5694,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.184296+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:12.184296+0000","last_peered":"2024-04-26T02:28:12.184296+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:12.184296+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46284800,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":45200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5789","reported_seq":10031,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:15.431618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:15.431618+0000","last_peered":"2024-04-26T02:28:15.431618+0000","last_clean":"2024-04-26T02:28:15.431618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:15.431618+0000","last_undegraded":"2024-04-26T02:28:15.431618+0000","last_fullsized":"2024-04-26T02:28:15.431618+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.963474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47370764,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5925","reported_seq":5991,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.576484+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:15.576484+0000","last_peered":"2024-04-26T02:28:15.576484+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:15.576484+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48493068,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":47372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5717","reported_seq":5774,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.181439+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:12.181439+0000","last_peered":"2024-04-26T02:28:12.181439+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:12.181439+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46760984,"num_objects":5709,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5714,"num_write_kb":45689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3705}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5676","reported_seq":5756,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.615137+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:13.615137+0000","last_peered":"2024-04-26T02:28:13.615137+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:13.615137+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46477836,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5716","reported_seq":5774,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.277313+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:12.277313+0000","last_peered":"2024-04-26T02:28:12.277313+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:12.277313+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46779154,"num_objects":5711,"num_object_clones":0,"num_object_copies":22844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":45692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5675","reported_seq":5712,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.221796+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:12.221796+0000","last_peered":"2024-04-26T02:28:12.221796+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:12.221796+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46479622,"num_objects":5674,"num_object_clones":0,"num_object_copies":22696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5756","reported_seq":5831,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.571316+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:13.571316+0000","last_peered":"2024-04-26T02:28:13.571316+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:13.571316+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47143174,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":46039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'5920","reported_seq":5973,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:12.115576+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:12.115576+0000","last_peered":"2024-04-26T02:28:12.115576+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:12.115576+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48460300,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":47340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3845}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5808","reported_seq":8340,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:13.747357+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:13.747357+0000","last_peered":"2024-04-26T02:28:13.747357+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:13.747357+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47550988,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1228,"num_bytes_recovered":10043392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4978}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5783","reported_seq":5848,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.539814+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:15.539814+0000","last_peered":"2024-04-26T02:28:15.539814+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:15.539814+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329804,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":46236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751934632,"num_objects":91795,"num_object_clones":0,"num_object_copies":367180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91822,"num_write_kb":734458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3337,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1383067648,"data_stored":1383067648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91849,"ondisk_log_size":91849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708544,"kb_used_data":365880,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93663296,"statfs":{"total":96636764160,"available":95911215104,"internally_reserved":0,"allocated":374661120,"data_stored":374475998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,7,9,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709436,"kb_used_data":367284,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93662404,"statfs":{"total":96636764160,"available":95910301696,"internally_reserved":0,"allocated":376098816,"data_stored":375913694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478500,"kb_used_data":248348,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93893340,"statfs":{"total":96636764160,"available":96146780160,"internally_reserved":0,"allocated":254308352,"data_stored":254123230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711424,"kb_used_data":368824,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93660416,"statfs":{"total":96636764160,"available":95908265984,"internally_reserved":0,"allocated":377675776,"data_stored":377490654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378019840,"data_stored":378019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254423040,"data_stored":254423040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376115200,"data_stored":376115200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374509568,"data_stored":374509568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5609", 'reported_seq': 5650, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.265922+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:12.265922+0000', 'last_peered': '2024-04-26T02:28:12.265922+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:12.265922+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5609, 'log_dups_size': 0, 'ondisk_log_size': 5609, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45948928, 'num_objects': 5609, 'num_object_clones': 0, 'num_object_copies': 22436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5609, 'num_write_kb': 44872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5820", 'reported_seq': 5886, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:15.506878+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:15.506878+0000', 'last_peered': '2024-04-26T02:28:15.506878+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:15.506878+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5820, 'log_dups_size': 0, 'ondisk_log_size': 5820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47641100, 'num_objects': 5816, 'num_object_clones': 0, 'num_object_copies': 23264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5818, 'num_write_kb': 46540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5683", 'reported_seq': 5762, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:13.753114+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:13.753114+0000', 'last_peered': '2024-04-26T02:28:13.753114+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:13.753114+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5683, 'log_dups_size': 0, 'ondisk_log_size': 5683, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46528774, 'num_objects': 5680, 'num_object_clones': 0, 'num_object_copies': 22720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5682, 'num_write_kb': 45446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3656}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5654", 'reported_seq': 5716, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:15.626525+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:15.626525+0000', 'last_peered': '2024-04-26T02:28:15.626525+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:15.626525+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5654, 'log_dups_size': 0, 'ondisk_log_size': 5654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46281228, 'num_objects': 5650, 'num_object_clones': 0, 'num_object_copies': 22600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5652, 'num_write_kb': 45205, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3682}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5668", 'reported_seq': 5724, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.219697+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:12.219697+0000', 'last_peered': '2024-04-26T02:28:12.219697+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:12.219697+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5668, 'log_dups_size': 0, 'ondisk_log_size': 5668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46404108, 'num_objects': 5665, 'num_object_clones': 0, 'num_object_copies': 22660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5666, 'num_write_kb': 45325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3654}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5650", 'reported_seq': 5694, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.184296+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:12.184296+0000', 'last_peered': '2024-04-26T02:28:12.184296+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:12.184296+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5650, 'log_dups_size': 0, 'ondisk_log_size': 5650, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46284800, 'num_objects': 5650, 'num_object_clones': 0, 'num_object_copies': 22600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5650, 'num_write_kb': 45200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'5925", 'reported_seq': 5991, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:15.576484+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:15.576484+0000', 'last_peered': '2024-04-26T02:28:15.576484+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:15.576484+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5925, 'log_dups_size': 0, 'ondisk_log_size': 5925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48493068, 'num_objects': 5920, 'num_object_clones': 0, 'num_object_copies': 23680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5923, 'num_write_kb': 47372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5717", 'reported_seq': 5774, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.181439+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:12.181439+0000', 'last_peered': '2024-04-26T02:28:12.181439+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:12.181439+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5717, 'log_dups_size': 0, 'ondisk_log_size': 5717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46760984, 'num_objects': 5709, 'num_object_clones': 0, 'num_object_copies': 22836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5714, 'num_write_kb': 45689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3705}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.006 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5676", 'reported_seq': 5756, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:13.615137+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:13.615137+0000', 'last_peered': '2024-04-26T02:28:13.615137+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:13.615137+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5676, 'log_dups_size': 0, 'ondisk_log_size': 5676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46477836, 'num_objects': 5674, 'num_object_clones': 0, 'num_object_copies': 22696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5674, 'num_write_kb': 45390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3637}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5716", 'reported_seq': 5774, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.277313+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:12.277313+0000', 'last_peered': '2024-04-26T02:28:12.277313+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:12.277313+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5716, 'log_dups_size': 0, 'ondisk_log_size': 5716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46779154, 'num_objects': 5711, 'num_object_clones': 0, 'num_object_copies': 22844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5713, 'num_write_kb': 45692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3694}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5675", 'reported_seq': 5712, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.221796+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:12.221796+0000', 'last_peered': '2024-04-26T02:28:12.221796+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:12.221796+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5675, 'log_dups_size': 0, 'ondisk_log_size': 5675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46479622, 'num_objects': 5674, 'num_object_clones': 0, 'num_object_copies': 22696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5674, 'num_write_kb': 45391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5756", 'reported_seq': 5831, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:13.571316+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:13.571316+0000', 'last_peered': '2024-04-26T02:28:13.571316+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:13.571316+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5756, 'log_dups_size': 0, 'ondisk_log_size': 5756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47143174, 'num_objects': 5755, 'num_object_clones': 0, 'num_object_copies': 23020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2112, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5755, 'num_write_kb': 46039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'5920", 'reported_seq': 5973, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:12.115576+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:12.115576+0000', 'last_peered': '2024-04-26T02:28:12.115576+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:12.115576+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5920, 'log_dups_size': 0, 'ondisk_log_size': 5920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48460300, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 23664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5918, 'num_write_kb': 47340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5808", 'reported_seq': 8340, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:13.747357+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:13.747357+0000', 'last_peered': '2024-04-26T02:28:13.747357+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:13.747357+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5808, 'log_dups_size': 0, 'ondisk_log_size': 5808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47550988, 'num_objects': 5805, 'num_object_clones': 0, 'num_object_copies': 23220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 827, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5806, 'num_write_kb': 46445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1228, 'num_bytes_recovered': 10043392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5783", 'reported_seq': 5848, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:15.539814+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:15.539814+0000', 'last_peered': '2024-04-26T02:28:15.539814+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:15.539814+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5783, 'log_dups_size': 0, 'ondisk_log_size': 5783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47329804, 'num_objects': 5778, 'num_object_clones': 0, 'num_object_copies': 23112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5781, 'num_write_kb': 46236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3714}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:18.008 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:18.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 237 20 93083 93063 3.06629 2.04688 0.051174 0.050943 2024-04-26T02:28:19.630 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:19.630 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29430/369488 objects degraded (7.965%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":296,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92372,"data_bytes":756661416,"bytes_used":2688909312,"bytes_avail":383858147328,"bytes_total":386547056640,"degraded_objects":29430,"degraded_total":369488,"degraded_ratio":0.079650759970553847,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":100325,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"write_bytes_sec":2169627,"read_op_per_sec":0,"write_op_per_sec":264},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:19.696 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:19.696 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:19.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 238 20 93308 93288 3.06079 1.75781 0.0474721 0.0510324 2024-04-26T02:28:20.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout: 239 20 93549 93529 3.05587 1.88281 0.0531791 0.0511165 2024-04-26T02:28:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:21.200 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:21.320 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-04-26T02:28:20.532342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759315624,"num_objects":92696,"num_object_clones":0,"num_object_copies":370784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92723,"num_write_kb":741666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":27754496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92750,"ondisk_log_size":92750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2635176,"kb_used_data":1365128,"kb_used_omap":14,"kb_used_meta":1270001,"kb_avail":374852184,"statfs":{"total":386547056640,"available":383848636416,"internally_reserved":0,"allocated":1397891072,"data_stored":1397150584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1300481587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,4,13,4,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23756800,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2900,"num_write_kb":23200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":1859584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.f","version":"34'5695","reported_seq":5737,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.245756+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:17.245756+0000","last_peered":"2024-04-26T02:28:17.245756+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:17.245756+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46653440,"num_objects":5695,"num_object_clones":0,"num_object_copies":22780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":45560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5820","reported_seq":5886,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.506878+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:15.506878+0000","last_peered":"2024-04-26T02:28:15.506878+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:15.506878+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47641100,"num_objects":5816,"num_object_clones":0,"num_object_copies":23264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":46540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5776","reported_seq":5854,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:18.808334+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:18.808334+0000","last_peered":"2024-04-26T02:28:18.808334+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:18.808334+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47290630,"num_objects":5773,"num_object_clones":0,"num_object_copies":23092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":46190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3749}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5654","reported_seq":5716,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.626525+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:15.626525+0000","last_peered":"2024-04-26T02:28:15.626525+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:15.626525+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46281228,"num_objects":5650,"num_object_clones":0,"num_object_copies":22600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":45205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5758","reported_seq":5814,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.345584+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:17.345584+0000","last_peered":"2024-04-26T02:28:17.345584+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:17.345584+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47141388,"num_objects":5755,"num_object_clones":0,"num_object_copies":23020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":46045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5727","reported_seq":5771,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.243377+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:17.243377+0000","last_peered":"2024-04-26T02:28:17.243377+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:17.243377+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46915584,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3676}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5789","reported_seq":10031,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:15.431618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:15.431618+0000","last_peered":"2024-04-26T02:28:15.431618+0000","last_clean":"2024-04-26T02:28:15.431618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:15.431618+0000","last_undegraded":"2024-04-26T02:28:15.431618+0000","last_fullsized":"2024-04-26T02:28:15.431618+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.963474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47370764,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5925","reported_seq":5991,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.576484+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:15.576484+0000","last_peered":"2024-04-26T02:28:15.576484+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:15.576484+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48493068,"num_objects":5920,"num_object_clones":0,"num_object_copies":23680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":47372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5806","reported_seq":5860,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.368463+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:17.368463+0000","last_peered":"2024-04-26T02:28:17.368463+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:17.368463+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47490072,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5738","reported_seq":5818,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:18.792384+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:18.792384+0000","last_peered":"2024-04-26T02:28:18.792384+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:18.792384+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46985740,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5736,"num_write_kb":45886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5799","reported_seq":5857,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.266452+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:17.266452+0000","last_peered":"2024-04-26T02:28:17.266452+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:17.266452+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459090,"num_objects":5794,"num_object_clones":0,"num_object_copies":23176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":46356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5745","reported_seq":5784,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.258013+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:17.258013+0000","last_peered":"2024-04-26T02:28:17.258013+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:17.258013+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47053062,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5845","reported_seq":5920,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:18.756551+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:18.756551+0000","last_peered":"2024-04-26T02:28:18.756551+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:18.756551+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47872262,"num_objects":5844,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":46751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3733}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6002","reported_seq":6055,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:17.124816+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:17.124816+0000","last_peered":"2024-04-26T02:28:17.124816+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:17.124816+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49132044,"num_objects":5998,"num_object_clones":0,"num_object_copies":23992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6000,"num_write_kb":47996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3927}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5888","reported_seq":8584,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:18.811671+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:18.811671+0000","last_peered":"2024-04-26T02:28:18.811671+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:18.811671+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48206348,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":47085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1310,"num_bytes_recovered":10698752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5783","reported_seq":5848,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:15.539814+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:15.539814+0000","last_peered":"2024-04-26T02:28:15.539814+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:15.539814+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329804,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":46236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3714}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759315624,"num_objects":92696,"num_object_clones":0,"num_object_copies":370784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92723,"num_write_kb":741666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3419,"num_bytes_recovered":27754496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1399050240,"data_stored":1399050240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92750,"ondisk_log_size":92750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716996,"kb_used_data":370172,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93654844,"statfs":{"total":96636764160,"available":95902560256,"internally_reserved":0,"allocated":379056128,"data_stored":378871006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":355135184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718724,"kb_used_data":372412,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93653116,"statfs":{"total":96636764160,"available":95900790784,"internally_reserved":0,"allocated":381349888,"data_stored":381164766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488032,"kb_used_data":253720,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93883808,"statfs":{"total":96636764160,"available":96137019392,"internally_reserved":0,"allocated":259809280,"data_stored":259624158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711424,"kb_used_data":368824,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93660416,"statfs":{"total":96636764160,"available":95908265984,"internally_reserved":0,"allocated":377675776,"data_stored":377490654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350809807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378019840,"data_stored":378019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259952640,"data_stored":259952640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381313024,"data_stored":381313024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379764736,"data_stored":379764736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:21.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Total time run: 240.05 2024-04-26T02:28:21.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Total writes made: 93736 2024-04-26T02:28:21.790 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Write size: 8192 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Object size: 8192 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Bandwidth (MB/sec): 3.05067 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Stddev Bandwidth: 1.89346 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Max bandwidth (MB/sec): 11.375 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Average IOPS: 390 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Stddev IOPS: 242.366 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Max IOPS: 1456 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Min IOPS: 0 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Average Latency(s): 0.0512102 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Stddev Latency(s): 0.053134 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Max latency(s): 2.17928 2024-04-26T02:28:21.791 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Min latency(s): 0.0028054 2024-04-26T02:28:21.802 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-04-26T02:28:24.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:25.497 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:25.498 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-04-26T02:28:24.535906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":766123200,"num_objects":93528,"num_object_clones":0,"num_object_copies":374112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93907,"num_write_kb":749723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3560,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93934,"ondisk_log_size":93934,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2621392,"kb_used_data":1380440,"kb_used_omap":14,"kb_used_meta":1240881,"kb_avail":374865968,"statfs":{"total":386547056640,"available":383862751232,"internally_reserved":0,"allocated":1413570560,"data_stored":1412830072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1270662707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,9,13,20,19,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19808280,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2771,"num_write_kb":20753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":2367488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005710"},"pg_stats":[{"pgid":"1.f","version":"34'5768","reported_seq":5810,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.161674+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:22.161674+0000","last_peered":"2024-04-26T02:28:22.161674+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:22.161674+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47153152,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5901","reported_seq":5967,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:20.678592+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:20.678592+0000","last_peered":"2024-04-26T02:28:20.678592+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:20.678592+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48304652,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5899,"num_write_kb":47188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5844","reported_seq":5926,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.797139+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:23.797139+0000","last_peered":"2024-04-26T02:28:23.797139+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:23.797139+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47397150,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5843,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5736","reported_seq":5800,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:20.669296+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:20.669296+0000","last_peered":"2024-04-26T02:28:20.669296+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:20.669296+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46952972,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":45861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5808","reported_seq":5865,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.121934+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:22.121934+0000","last_peered":"2024-04-26T02:28:22.121934+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:22.121934+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47419916,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5805","reported_seq":5849,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.072801+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:22.072801+0000","last_peered":"2024-04-26T02:28:22.072801+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:22.072801+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47472640,"num_objects":5795,"num_object_clones":0,"num_object_copies":23180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5875","reported_seq":10117,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:20.647588+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:20.647588+0000","last_peered":"2024-04-26T02:28:20.647588+0000","last_clean":"2024-04-26T02:28:20.647588+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:20.647588+0000","last_undegraded":"2024-04-26T02:28:20.647588+0000","last_fullsized":"2024-04-26T02:28:20.647588+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:24.004367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48075276,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'5990","reported_seq":6056,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:20.693945+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:20.693945+0000","last_peered":"2024-04-26T02:28:20.693945+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:20.693945+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49025548,"num_objects":5985,"num_object_clones":0,"num_object_copies":23940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5878","reported_seq":5935,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.110823+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:22.110823+0000","last_peered":"2024-04-26T02:28:22.110823+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:22.110823+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5878,"log_dups_size":0,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48014360,"num_objects":5862,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5875,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5818","reported_seq":5902,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.753453+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:23.753453+0000","last_peered":"2024-04-26T02:28:23.753453+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:23.753453+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47067660,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5869","reported_seq":5927,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.027449+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:22.027449+0000","last_peered":"2024-04-26T02:28:22.027449+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:22.027449+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47917842,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5805","reported_seq":5844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.125163+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:22.125163+0000","last_peered":"2024-04-26T02:28:22.125163+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:22.125163+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47446278,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5926","reported_seq":6010,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.868879+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:23.868879+0000","last_peered":"2024-04-26T02:28:23.868879+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:23.868879+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47945990,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6066","reported_seq":6118,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.128315+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:22.128315+0000","last_peered":"2024-04-26T02:28:22.128315+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:22.128315+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49607180,"num_objects":6056,"num_object_clones":0,"num_object_copies":24224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6064,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5968","reported_seq":8931,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.847251+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:23.847251+0000","last_peered":"2024-04-26T02:28:23.847251+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:23.847251+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48222732,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1443,"num_bytes_recovered":11788288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5877","reported_seq":5939,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:20.702544+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:20.702544+0000","last_peered":"2024-04-26T02:28:20.702544+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:20.702544+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48099852,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5875,"num_write_kb":46988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":766123200,"num_objects":93528,"num_object_clones":0,"num_object_copies":374112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93907,"num_write_kb":749723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3560,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1412816896,"data_stored":1412816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93934,"ondisk_log_size":93934,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":375032,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":384032768,"data_stored":383847646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,2,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707280,"kb_used_data":373424,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93664560,"statfs":{"total":96636764160,"available":95912509440,"internally_reserved":0,"allocated":382386176,"data_stored":382201054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,7,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484288,"kb_used_data":258296,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93887552,"statfs":{"total":96636764160,"available":96140853248,"internally_reserved":0,"allocated":264495104,"data_stored":264309982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720448,"kb_used_data":373688,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93651392,"statfs":{"total":96636764160,"available":95899025408,"internally_reserved":0,"allocated":382656512,"data_stored":382471390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383078400,"data_stored":383078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264273920,"data_stored":264273920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381947904,"data_stored":381947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383516672,"data_stored":383516672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:25.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:26.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:26.654 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:26.768 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-26T02:28:26.537715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":764075200,"num_objects":93278,"num_object_clones":0,"num_object_copies":373112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94279,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3561,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94306,"ondisk_log_size":94306,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625428,"kb_used_data":1378908,"kb_used_omap":14,"kb_used_meta":1246449,"kb_avail":374861932,"statfs":{"total":386547056640,"available":383858618368,"internally_reserved":0,"allocated":1412001792,"data_stored":1411261304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1276364339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,2,8,12,17,19,5,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15163416,"num_objects":1852,"num_object_clones":0,"num_object_copies":7408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2826,"num_write_kb":18705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006973"},"pg_stats":[{"pgid":"1.f","version":"34'5768","reported_seq":5810,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.161674+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:22.161674+0000","last_peered":"2024-04-26T02:28:22.161674+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:22.161674+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47153152,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5989","reported_seq":6055,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.656803+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:25.656803+0000","last_peered":"2024-04-26T02:28:25.656803+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:25.656803+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862284,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5844","reported_seq":5926,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.797139+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:23.797139+0000","last_peered":"2024-04-26T02:28:23.797139+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:23.797139+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47397150,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5843,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5803","reported_seq":5867,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.728017+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:25.728017+0000","last_peered":"2024-04-26T02:28:25.728017+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:25.728017+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46502412,"num_objects":5677,"num_object_clones":0,"num_object_copies":22708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5808","reported_seq":5865,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.121934+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:22.121934+0000","last_peered":"2024-04-26T02:28:22.121934+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:22.121934+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47419916,"num_objects":5789,"num_object_clones":0,"num_object_copies":23156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5805","reported_seq":5849,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.072801+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:22.072801+0000","last_peered":"2024-04-26T02:28:22.072801+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:22.072801+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47472640,"num_objects":5795,"num_object_clones":0,"num_object_copies":23180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5937","reported_seq":10180,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:25.482379+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:25.482379+0000","last_peered":"2024-04-26T02:28:25.482379+0000","last_clean":"2024-04-26T02:28:25.482379+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:25.482379+0000","last_undegraded":"2024-04-26T02:28:25.482379+0000","last_fullsized":"2024-04-26T02:28:25.482379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:29.041145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47682060,"num_objects":5821,"num_object_clones":0,"num_object_copies":23284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5935,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6051","reported_seq":6118,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.688196+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:25.688196+0000","last_peered":"2024-04-26T02:28:25.688196+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:25.688196+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48640524,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3797}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5878","reported_seq":5935,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.110823+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:22.110823+0000","last_peered":"2024-04-26T02:28:22.110823+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:22.110823+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5878,"log_dups_size":0,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48014360,"num_objects":5862,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5875,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5818","reported_seq":5902,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.753453+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:23.753453+0000","last_peered":"2024-04-26T02:28:23.753453+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:23.753453+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47067660,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5869","reported_seq":5927,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.027449+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:22.027449+0000","last_peered":"2024-04-26T02:28:22.027449+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:22.027449+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47917842,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5805","reported_seq":5844,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.125163+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:22.125163+0000","last_peered":"2024-04-26T02:28:22.125163+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:22.125163+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47446278,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5926","reported_seq":6010,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.868879+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:23.868879+0000","last_peered":"2024-04-26T02:28:23.868879+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:23.868879+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47945990,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6066","reported_seq":6118,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:22.128315+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:22.128315+0000","last_peered":"2024-04-26T02:28:22.128315+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:22.128315+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49607180,"num_objects":6056,"num_object_clones":0,"num_object_copies":24224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6064,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5968","reported_seq":8931,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.847251+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:23.847251+0000","last_peered":"2024-04-26T02:28:23.847251+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:23.847251+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48222732,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1443,"num_bytes_recovered":11788288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5971","reported_seq":6036,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.694447+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:25.694447+0000","last_peered":"2024-04-26T02:28:25.694447+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:25.694447+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723020,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":764075200,"num_objects":93278,"num_object_clones":0,"num_object_copies":373112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94279,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3561,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1409826816,"data_stored":1409826816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94306,"ondisk_log_size":94306,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":375032,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":384032768,"data_stored":383847646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":342355664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,2,2,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707280,"kb_used_data":373424,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93664560,"statfs":{"total":96636764160,"available":95912509440,"internally_reserved":0,"allocated":382386176,"data_stored":382201054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,7,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484288,"kb_used_data":258296,"kb_used_omap":4,"kb_used_meta":225979,"kb_avail":93887552,"statfs":{"total":96636764160,"available":96140853248,"internally_reserved":0,"allocated":264495104,"data_stored":264309982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231403215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724484,"kb_used_data":372156,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93647356,"statfs":{"total":96636764160,"available":95894892544,"internally_reserved":0,"allocated":381087744,"data_stored":380902622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380088320,"data_stored":380088320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264273920,"data_stored":264273920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381947904,"data_stored":381947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383516672,"data_stored":383516672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5768", 'reported_seq': 5810, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.161674+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:22.161674+0000', 'last_peered': '2024-04-26T02:28:22.161674+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:22.161674+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5768, 'log_dups_size': 0, 'ondisk_log_size': 5768, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47153152, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 23024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5768, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3739}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'5989", 'reported_seq': 6055, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:25.656803+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:25.656803+0000', 'last_peered': '2024-04-26T02:28:25.656803+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:25.656803+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5989, 'log_dups_size': 0, 'ondisk_log_size': 5989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47862284, 'num_objects': 5843, 'num_object_clones': 0, 'num_object_copies': 23372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5987, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5844", 'reported_seq': 5926, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:23.797139+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:23.797139+0000', 'last_peered': '2024-04-26T02:28:23.797139+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:23.797139+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5844, 'log_dups_size': 0, 'ondisk_log_size': 5844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47397150, 'num_objects': 5787, 'num_object_clones': 0, 'num_object_copies': 23148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5787, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 5843, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3764}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5803", 'reported_seq': 5867, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:25.728017+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:25.728017+0000', 'last_peered': '2024-04-26T02:28:25.728017+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:25.728017+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46502412, 'num_objects': 5677, 'num_object_clones': 0, 'num_object_copies': 22708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:26.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5808", 'reported_seq': 5865, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.121934+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:22.121934+0000', 'last_peered': '2024-04-26T02:28:22.121934+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:22.121934+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5808, 'log_dups_size': 0, 'ondisk_log_size': 5808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47419916, 'num_objects': 5789, 'num_object_clones': 0, 'num_object_copies': 23156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5806, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3778}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5805", 'reported_seq': 5849, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.072801+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:22.072801+0000', 'last_peered': '2024-04-26T02:28:22.072801+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:22.072801+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5805, 'log_dups_size': 0, 'ondisk_log_size': 5805, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47472640, 'num_objects': 5795, 'num_object_clones': 0, 'num_object_copies': 23180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5805, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3744}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'6051", 'reported_seq': 6118, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:25.688196+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:25.688196+0000', 'last_peered': '2024-04-26T02:28:25.688196+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:25.688196+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6051, 'log_dups_size': 0, 'ondisk_log_size': 6051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48640524, 'num_objects': 5938, 'num_object_clones': 0, 'num_object_copies': 23752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6049, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'5878", 'reported_seq': 5935, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.110823+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:22.110823+0000', 'last_peered': '2024-04-26T02:28:22.110823+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:22.110823+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5878, 'log_dups_size': 0, 'ondisk_log_size': 5878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48014360, 'num_objects': 5862, 'num_object_clones': 0, 'num_object_copies': 23448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5875, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3858}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5818", 'reported_seq': 5902, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:23.753453+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:23.753453+0000', 'last_peered': '2024-04-26T02:28:23.753453+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:23.753453+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5818, 'log_dups_size': 0, 'ondisk_log_size': 5818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47067660, 'num_objects': 5746, 'num_object_clones': 0, 'num_object_copies': 22984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5816, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3711}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:26.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'5869", 'reported_seq': 5927, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.027449+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:22.027449+0000', 'last_peered': '2024-04-26T02:28:22.027449+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:22.027449+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5869, 'log_dups_size': 0, 'ondisk_log_size': 5869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47917842, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5866, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3833}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5805", 'reported_seq': 5844, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.125163+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:22.125163+0000', 'last_peered': '2024-04-26T02:28:22.125163+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:22.125163+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5805, 'log_dups_size': 0, 'ondisk_log_size': 5805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47446278, 'num_objects': 5792, 'num_object_clones': 0, 'num_object_copies': 23168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5804, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'5926", 'reported_seq': 6010, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:23.868879+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:23.868879+0000', 'last_peered': '2024-04-26T02:28:23.868879+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:23.868879+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5926, 'log_dups_size': 0, 'ondisk_log_size': 5926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47945990, 'num_objects': 5853, 'num_object_clones': 0, 'num_object_copies': 23412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2107, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5925, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'6066", 'reported_seq': 6118, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:22.128315+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:22.128315+0000', 'last_peered': '2024-04-26T02:28:22.128315+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:22.128315+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6066, 'log_dups_size': 0, 'ondisk_log_size': 6066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49607180, 'num_objects': 6056, 'num_object_clones': 0, 'num_object_copies': 24224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6064, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'5968", 'reported_seq': 8931, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:23.847251+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:23.847251+0000', 'last_peered': '2024-04-26T02:28:23.847251+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:23.847251+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5968, 'log_dups_size': 0, 'ondisk_log_size': 5968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48222732, 'num_objects': 5887, 'num_object_clones': 0, 'num_object_copies': 23548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 614, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5966, 'num_write_kb': 47413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1443, 'num_bytes_recovered': 11788288, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 5273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:26.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'5971", 'reported_seq': 6036, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:25.694447+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:25.694447+0000', 'last_peered': '2024-04-26T02:28:25.694447+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:25.694447+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5971, 'log_dups_size': 0, 'ondisk_log_size': 5971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47723020, 'num_objects': 5826, 'num_object_clones': 0, 'num_object_copies': 23304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5969, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:26.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:28.331 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:28.331 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29208/373112 objects degraded (7.828%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":304,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93278,"data_bytes":764075200,"bytes_used":2688438272,"bytes_avail":383858618368,"bytes_total":386547056640,"degraded_objects":29208,"degraded_total":373112,"degraded_ratio":0.078282124402324238,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":145323,"recovering_keys_per_sec":0,"num_objects_recovered":224,"num_bytes_recovered":1744896,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1595233,"read_op_per_sec":0,"write_op_per_sec":235},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:28.408 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:28.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:29.891 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:29.935 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-04-26T02:28:28.538876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":759626944,"num_objects":92735,"num_object_clones":0,"num_object_copies":370940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94822,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94849,"ondisk_log_size":94849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2650712,"kb_used_data":1373656,"kb_used_omap":14,"kb_used_meta":1276977,"kb_avail":374836648,"statfs":{"total":386547056640,"available":383832727552,"internally_reserved":0,"allocated":1406623744,"data_stored":1405883256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1307625011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,8,9,11,16,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7692312,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3000,"num_write_kb":15753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007596"},"pg_stats":[{"pgid":"1.f","version":"34'5844","reported_seq":5892,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.304256+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:27.304256+0000","last_peered":"2024-04-26T02:28:27.304256+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:27.304256+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'5989","reported_seq":6055,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.656803+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:25.656803+0000","last_peered":"2024-04-26T02:28:25.656803+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:25.656803+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47862284,"num_objects":5843,"num_object_clones":0,"num_object_copies":23372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5844","reported_seq":5926,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.797139+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:23.797139+0000","last_peered":"2024-04-26T02:28:23.797139+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:23.797139+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47397150,"num_objects":5787,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5843,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3764}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5803","reported_seq":5867,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.728017+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:25.728017+0000","last_peered":"2024-04-26T02:28:25.728017+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:25.728017+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46502412,"num_objects":5677,"num_object_clones":0,"num_object_copies":22708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5887","reported_seq":5950,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.217279+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:27.217279+0000","last_peered":"2024-04-26T02:28:27.217279+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:27.217279+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46772748,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5901","reported_seq":5945,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.297084+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:27.297084+0000","last_peered":"2024-04-26T02:28:27.297084+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:27.297084+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686208,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5901,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'5937","reported_seq":10180,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:25.482379+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:25.482379+0000","last_peered":"2024-04-26T02:28:25.482379+0000","last_clean":"2024-04-26T02:28:25.482379+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:25.482379+0000","last_undegraded":"2024-04-26T02:28:25.482379+0000","last_fullsized":"2024-04-26T02:28:25.482379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:29.041145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47682060,"num_objects":5821,"num_object_clones":0,"num_object_copies":23284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5935,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6051","reported_seq":6118,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.688196+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:25.688196+0000","last_peered":"2024-04-26T02:28:25.688196+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:25.688196+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48640524,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3797}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'5949","reported_seq":6005,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.361471+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:27.361471+0000","last_peered":"2024-04-26T02:28:27.361471+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:27.361471+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5949,"log_dups_size":0,"ondisk_log_size":5949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47432728,"num_objects":5791,"num_object_clones":0,"num_object_copies":23164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3787}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5818","reported_seq":5902,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.753453+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:23.753453+0000","last_peered":"2024-04-26T02:28:23.753453+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:23.753453+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47067660,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'5941","reported_seq":5998,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.397898+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:27.397898+0000","last_peered":"2024-04-26T02:28:27.397898+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:27.397898+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47328018,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5877","reported_seq":5918,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.242072+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:27.242072+0000","last_peered":"2024-04-26T02:28:27.242072+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:27.242072+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46856454,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3733}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'5926","reported_seq":6010,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.868879+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:23.868879+0000","last_peered":"2024-04-26T02:28:23.868879+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:23.868879+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47945990,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6143","reported_seq":6198,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:27.387954+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:27.387954+0000","last_peered":"2024-04-26T02:28:27.387954+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:27.387954+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6143,"log_dups_size":0,"ondisk_log_size":6143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48976396,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3909}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'5968","reported_seq":8931,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:23.847251+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:23.847251+0000","last_peered":"2024-04-26T02:28:23.847251+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:23.847251+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48222732,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1443,"num_bytes_recovered":11788288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'5971","reported_seq":6036,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:25.694447+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:25.694447+0000","last_peered":"2024-04-26T02:28:25.694447+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:25.694447+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723020,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":759626944,"num_objects":92735,"num_object_clones":0,"num_object_copies":370940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94822,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":28844032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1404497920,"data_stored":1404497920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94849,"ondisk_log_size":94849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730620,"kb_used_data":369900,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93641220,"statfs":{"total":96636764160,"available":95888609280,"internally_reserved":0,"allocated":378777600,"data_stored":378592478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,2,4,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707280,"kb_used_data":373424,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93664560,"statfs":{"total":96636764160,"available":95912509440,"internally_reserved":0,"allocated":382386176,"data_stored":382201054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,7,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488328,"kb_used_data":258176,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93883512,"statfs":{"total":96636764160,"available":96136716288,"internally_reserved":0,"allocated":264372224,"data_stored":264187102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724484,"kb_used_data":372156,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93647356,"statfs":{"total":96636764160,"available":95894892544,"internally_reserved":0,"allocated":381087744,"data_stored":380902622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380088320,"data_stored":380088320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264052736,"data_stored":264052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381947904,"data_stored":381947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":378408960,"data_stored":378408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:32.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:34.422 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:34.422 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:34.559 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-04-26T02:28:32.539915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749395136,"num_objects":91486,"num_object_clones":0,"num_object_copies":365944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96071,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3708,"num_bytes_recovered":29573120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96098,"ondisk_log_size":96098,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670380,"kb_used_data":1358652,"kb_used_omap":14,"kb_used_meta":1311665,"kb_avail":374816980,"statfs":{"total":386547056640,"available":383812587520,"internally_reserved":0,"allocated":1391259648,"data_stored":1390519160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1343145523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,16,5,8,15,11,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9920488,"num_objects":-1210,"num_object_clones":0,"num_object_copies":-4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3348,"num_write_kb":8545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007573"},"pg_stats":[{"pgid":"1.f","version":"34'5918","reported_seq":5976,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.289165+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:32.289165+0000","last_peered":"2024-04-26T02:28:32.289165+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:32.289165+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45924352,"num_objects":5606,"num_object_clones":0,"num_object_copies":22424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6068","reported_seq":6137,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.619000+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:30.619000+0000","last_peered":"2024-04-26T02:28:30.619000+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:30.619000+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215116,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5918","reported_seq":6002,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:28.859509+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:28.859509+0000","last_peered":"2024-04-26T02:28:28.859509+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:28.859509+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46790942,"num_objects":5713,"num_object_clones":0,"num_object_copies":22852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5917,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3692}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5886","reported_seq":5956,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.558732+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:30.558732+0000","last_peered":"2024-04-26T02:28:30.558732+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:30.558732+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45822476,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5962","reported_seq":6035,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.425099+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:32.425099+0000","last_peered":"2024-04-26T02:28:32.425099+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:32.425099+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46158348,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5992","reported_seq":6042,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.288946+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:32.288946+0000","last_peered":"2024-04-26T02:28:32.288946+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:32.288946+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45940736,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5992,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6027","reported_seq":10269,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:30.735741+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:30.735741+0000","last_peered":"2024-04-26T02:28:30.735741+0000","last_clean":"2024-04-26T02:28:30.735741+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:30.735741+0000","last_undegraded":"2024-04-26T02:28:30.735741+0000","last_fullsized":"2024-04-26T02:28:30.735741+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:34.992045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46944780,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6124","reported_seq":6197,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.540561+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:30.540561+0000","last_peered":"2024-04-26T02:28:30.540561+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:30.540561+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48042508,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6022","reported_seq":6088,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.354061+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:32.354061+0000","last_peered":"2024-04-26T02:28:32.354061+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:32.354061+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46834712,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5891","reported_seq":5987,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:28.900938+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:28.900938+0000","last_peered":"2024-04-26T02:28:28.900938+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:28.900938+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5891,"log_dups_size":0,"ondisk_log_size":5891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46469644,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5889,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6009","reported_seq":6070,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.116832+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:32.116832+0000","last_peered":"2024-04-26T02:28:32.116832+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:32.116832+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46770962,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5955","reported_seq":6006,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.437469+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:32.437469+0000","last_peered":"2024-04-26T02:28:32.437469+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:32.437469+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217478,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6002","reported_seq":6094,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:28.926488+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:28.926488+0000","last_peered":"2024-04-26T02:28:28.926488+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:28.926488+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323398,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6226","reported_seq":6289,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.311610+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:32.311610+0000","last_peered":"2024-04-26T02:28:32.311610+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:32.311610+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48296460,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6051","reported_seq":9192,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:28.928413+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:28.928413+0000","last_peered":"2024-04-26T02:28:28.928413+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:28.928413+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47542796,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6047","reported_seq":6122,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.643942+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:30.643942+0000","last_peered":"2024-04-26T02:28:30.643942+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:30.643942+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47100428,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6045,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749395136,"num_objects":91486,"num_object_clones":0,"num_object_copies":365944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96071,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3708,"num_bytes_recovered":29573120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388843008,"data_stored":1388843008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96098,"ondisk_log_size":96098,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729644,"kb_used_data":364768,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93642196,"statfs":{"total":96636764160,"available":95889608704,"internally_reserved":0,"allocated":373522432,"data_stored":373337310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,3,7,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728644,"kb_used_data":368436,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93643196,"statfs":{"total":96636764160,"available":95890632704,"internally_reserved":0,"allocated":377278464,"data_stored":377093342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488328,"kb_used_data":258176,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93883512,"statfs":{"total":96636764160,"available":96136716288,"internally_reserved":0,"allocated":264372224,"data_stored":264187102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723764,"kb_used_data":367272,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93648076,"statfs":{"total":96636764160,"available":95895629824,"internally_reserved":0,"allocated":376086528,"data_stored":375901406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,2,2,1,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374935552,"data_stored":374935552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264052736,"data_stored":264052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376782848,"data_stored":376782848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373071872,"data_stored":373071872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:34.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:35.503 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:35.503 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:35.579 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-04-26T02:28:34.540459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":746740928,"num_objects":91162,"num_object_clones":0,"num_object_copies":364648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96395,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3808,"num_bytes_recovered":30351360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96422,"ondisk_log_size":96422,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691188,"kb_used_data":1353096,"kb_used_omap":14,"kb_used_meta":1338033,"kb_avail":374796172,"statfs":{"total":386547056640,"available":383791280128,"internally_reserved":0,"allocated":1385570304,"data_stored":1384829816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1370146355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,20,4,11,11,13,0,2,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19103720,"num_objects":-2331,"num_object_clones":0,"num_object_copies":-9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2797,"num_write_kb":1857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":389,"num_bytes_recovered":2596864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005096"},"pg_stats":[{"pgid":"1.f","version":"34'5918","reported_seq":5976,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.289165+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:32.289165+0000","last_peered":"2024-04-26T02:28:32.289165+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:32.289165+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45924352,"num_objects":5606,"num_object_clones":0,"num_object_copies":22424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6068","reported_seq":6137,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.619000+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:30.619000+0000","last_peered":"2024-04-26T02:28:30.619000+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:30.619000+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215116,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5995","reported_seq":6087,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.017200+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:34.017200+0000","last_peered":"2024-04-26T02:28:34.017200+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:34.017200+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46160158,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5994,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3618}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5886","reported_seq":5956,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.558732+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:30.558732+0000","last_peered":"2024-04-26T02:28:30.558732+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:30.558732+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45822476,"num_objects":5594,"num_object_clones":0,"num_object_copies":22376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5962","reported_seq":6035,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.425099+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:32.425099+0000","last_peered":"2024-04-26T02:28:32.425099+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:32.425099+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46158348,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5992","reported_seq":6042,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.288946+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:32.288946+0000","last_peered":"2024-04-26T02:28:32.288946+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:32.288946+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45940736,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5992,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6027","reported_seq":10269,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:30.735741+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:30.735741+0000","last_peered":"2024-04-26T02:28:30.735741+0000","last_clean":"2024-04-26T02:28:30.735741+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:30.735741+0000","last_undegraded":"2024-04-26T02:28:30.735741+0000","last_fullsized":"2024-04-26T02:28:30.735741+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:34.992045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46944780,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6124","reported_seq":6197,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.540561+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:30.540561+0000","last_peered":"2024-04-26T02:28:30.540561+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:30.540561+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48042508,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6022","reported_seq":6088,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.354061+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:32.354061+0000","last_peered":"2024-04-26T02:28:32.354061+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:32.354061+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46834712,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5968","reported_seq":6064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:33.973777+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:33.973777+0000","last_peered":"2024-04-26T02:28:33.973777+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:33.973777+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45838860,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3567}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6009","reported_seq":6070,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.116832+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:32.116832+0000","last_peered":"2024-04-26T02:28:32.116832+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:32.116832+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46770962,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5955","reported_seq":6006,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.437469+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:32.437469+0000","last_peered":"2024-04-26T02:28:32.437469+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:32.437469+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217478,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6086","reported_seq":6181,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.057266+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:34.057266+0000","last_peered":"2024-04-26T02:28:34.057266+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:34.057266+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46635270,"num_objects":5693,"num_object_clones":0,"num_object_copies":22772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6226","reported_seq":6289,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.311610+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:32.311610+0000","last_peered":"2024-04-26T02:28:32.311610+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:32.311610+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48296460,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6137","reported_seq":9466,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.086814+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:34.086814+0000","last_peered":"2024-04-26T02:28:34.086814+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:34.086814+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838284,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13295616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6047","reported_seq":6122,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:30.643942+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:30.643942+0000","last_peered":"2024-04-26T02:28:30.643942+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:30.643942+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47100428,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6045,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":746740928,"num_objects":91162,"num_object_clones":0,"num_object_copies":364648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96395,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3808,"num_bytes_recovered":30351360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1383116800,"data_stored":1383116800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96422,"ondisk_log_size":96422,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729644,"kb_used_data":364768,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93642196,"statfs":{"total":96636764160,"available":95889608704,"internally_reserved":0,"allocated":373522432,"data_stored":373337310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,3,7,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727644,"kb_used_data":363280,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93644196,"statfs":{"total":96636764160,"available":95891656704,"internally_reserved":0,"allocated":371998720,"data_stored":371813598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,5,1,3,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510136,"kb_used_data":257776,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93861704,"statfs":{"total":96636764160,"available":96114384896,"internally_reserved":0,"allocated":263962624,"data_stored":263777502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723764,"kb_used_data":367272,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93648076,"statfs":{"total":96636764160,"available":95895629824,"internally_reserved":0,"allocated":376086528,"data_stored":375901406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,2,2,1,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374935552,"data_stored":374935552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263643136,"data_stored":263643136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371466240,"data_stored":371466240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373071872,"data_stored":373071872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'5918", 'reported_seq': 5976, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.289165+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:32.289165+0000', 'last_peered': '2024-04-26T02:28:32.289165+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:32.289165+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5918, 'log_dups_size': 0, 'ondisk_log_size': 5918, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45924352, 'num_objects': 5606, 'num_object_clones': 0, 'num_object_copies': 22424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5918, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'6068", 'reported_seq': 6137, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:30.619000+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:30.619000+0000', 'last_peered': '2024-04-26T02:28:30.619000+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:30.619000+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6068, 'log_dups_size': 0, 'ondisk_log_size': 6068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47215116, 'num_objects': 5764, 'num_object_clones': 0, 'num_object_copies': 23056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6066, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'5995", 'reported_seq': 6087, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:34.017200+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:34.017200+0000', 'last_peered': '2024-04-26T02:28:34.017200+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:34.017200+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5995, 'log_dups_size': 0, 'ondisk_log_size': 5995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46160158, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 5994, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3618}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:35.580 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'5886", 'reported_seq': 5956, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:30.558732+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:30.558732+0000', 'last_peered': '2024-04-26T02:28:30.558732+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:30.558732+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5886, 'log_dups_size': 0, 'ondisk_log_size': 5886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45822476, 'num_objects': 5594, 'num_object_clones': 0, 'num_object_copies': 22376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1967, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5884, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'5962", 'reported_seq': 6035, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.425099+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:32.425099+0000', 'last_peered': '2024-04-26T02:28:32.425099+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:32.425099+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5962, 'log_dups_size': 0, 'ondisk_log_size': 5962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46158348, 'num_objects': 5635, 'num_object_clones': 0, 'num_object_copies': 22540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5960, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'5992", 'reported_seq': 6042, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.288946+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:32.288946+0000', 'last_peered': '2024-04-26T02:28:32.288946+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:32.288946+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5992, 'log_dups_size': 0, 'ondisk_log_size': 5992, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45940736, 'num_objects': 5608, 'num_object_clones': 0, 'num_object_copies': 22432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5992, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'6124", 'reported_seq': 6197, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:30.540561+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:30.540561+0000', 'last_peered': '2024-04-26T02:28:30.540561+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:30.540561+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6124, 'log_dups_size': 0, 'ondisk_log_size': 6124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48042508, 'num_objects': 5865, 'num_object_clones': 0, 'num_object_copies': 23460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6122, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'6022", 'reported_seq': 6088, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.354061+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:32.354061+0000', 'last_peered': '2024-04-26T02:28:32.354061+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:32.354061+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6022, 'log_dups_size': 0, 'ondisk_log_size': 6022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46834712, 'num_objects': 5718, 'num_object_clones': 0, 'num_object_copies': 22872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2003, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6019, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3718}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:35.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'5968", 'reported_seq': 6064, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:33.973777+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:33.973777+0000', 'last_peered': '2024-04-26T02:28:33.973777+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:33.973777+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5968, 'log_dups_size': 0, 'ondisk_log_size': 5968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45838860, 'num_objects': 5596, 'num_object_clones': 0, 'num_object_copies': 22384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5966, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3567}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'6009", 'reported_seq': 6070, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.116832+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:32.116832+0000', 'last_peered': '2024-04-26T02:28:32.116832+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:32.116832+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6009, 'log_dups_size': 0, 'ondisk_log_size': 6009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46770962, 'num_objects': 5710, 'num_object_clones': 0, 'num_object_copies': 22840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6006, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3694}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'5955", 'reported_seq': 6006, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.437469+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:32.437469+0000', 'last_peered': '2024-04-26T02:28:32.437469+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:32.437469+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 5955, 'log_dups_size': 0, 'ondisk_log_size': 5955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46217478, 'num_objects': 5642, 'num_object_clones': 0, 'num_object_copies': 22568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1983, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5954, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3660}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'6086", 'reported_seq': 6181, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:34.057266+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:34.057266+0000', 'last_peered': '2024-04-26T02:28:34.057266+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:34.057266+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6086, 'log_dups_size': 0, 'ondisk_log_size': 6086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46635270, 'num_objects': 5693, 'num_object_clones': 0, 'num_object_copies': 22772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2102, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6085, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'6226", 'reported_seq': 6289, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:32.311610+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:32.311610+0000', 'last_peered': '2024-04-26T02:28:32.311610+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:32.311610+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6226, 'log_dups_size': 0, 'ondisk_log_size': 6226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48296460, 'num_objects': 5896, 'num_object_clones': 0, 'num_object_copies': 23584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2068, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6224, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:35.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'6137", 'reported_seq': 9466, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:34.086814+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:34.086814+0000', 'last_peered': '2024-04-26T02:28:34.086814+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:34.086814+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6137, 'log_dups_size': 0, 'ondisk_log_size': 6137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46838284, 'num_objects': 5718, 'num_object_clones': 0, 'num_object_copies': 22872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 430, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6135, 'num_write_kb': 47413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1627, 'num_bytes_recovered': 13295616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 5288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:35.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:35.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'6047", 'reported_seq': 6122, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:30.643942+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:30.643942+0000', 'last_peered': '2024-04-26T02:28:30.643942+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:30.643942+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6047, 'log_dups_size': 0, 'ondisk_log_size': 6047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47100428, 'num_objects': 5750, 'num_object_clones': 0, 'num_object_copies': 23000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6045, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:35.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:37.094 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28955/363040 objects degraded (7.976%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90760,"data_bytes":743447744,"bytes_used":2759053312,"bytes_avail":383788003328,"bytes_total":386547056640,"degraded_objects":28955,"degraded_total":363040,"degraded_ratio":0.07975705156456589,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":125557,"recovering_keys_per_sec":0,"num_objects_recovered":254,"num_bytes_recovered":1507328,"num_keys_recovered":0,"write_bytes_sec":41625,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:37.207 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:37.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:38.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:38.428 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:38.443 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-04-26T02:28:36.540945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":743447744,"num_objects":90760,"num_object_clones":0,"num_object_copies":363040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96797,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3814,"num_bytes_recovered":30351360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96824,"ondisk_log_size":96824,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694388,"kb_used_data":1347980,"kb_used_omap":14,"kb_used_meta":1346353,"kb_avail":374792972,"statfs":{"total":386547056640,"available":383788003328,"internally_reserved":0,"allocated":1380331520,"data_stored":1379591032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1378666035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,17,6,9,12,5,0,2,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22675456,"num_objects":-2768,"num_object_clones":0,"num_object_copies":-11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005039"},"pg_stats":[{"pgid":"1.f","version":"34'5918","reported_seq":5976,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.289165+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:32.289165+0000","last_peered":"2024-04-26T02:28:32.289165+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:32.289165+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45924352,"num_objects":5606,"num_object_clones":0,"num_object_copies":22424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5918,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6153","reported_seq":6227,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:35.742903+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:35.742903+0000","last_peered":"2024-04-26T02:28:35.742903+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:35.742903+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6153,"log_dups_size":0,"ondisk_log_size":6153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46518796,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6151,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'5995","reported_seq":6087,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.017200+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:34.017200+0000","last_peered":"2024-04-26T02:28:34.017200+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:34.017200+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5995,"log_dups_size":0,"ondisk_log_size":5995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46160158,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5994,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3618}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'5959","reported_seq":6029,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:35.668781+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:35.668781+0000","last_peered":"2024-04-26T02:28:35.668781+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:35.668781+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45224460,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'5962","reported_seq":6035,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.425099+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:32.425099+0000","last_peered":"2024-04-26T02:28:32.425099+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:32.425099+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46158348,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'5992","reported_seq":6042,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.288946+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:32.288946+0000","last_peered":"2024-04-26T02:28:32.288946+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:32.288946+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45940736,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5992,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6104","reported_seq":10347,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:35.722964+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:35.722964+0000","last_peered":"2024-04-26T02:28:35.722964+0000","last_clean":"2024-04-26T02:28:35.722964+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:35.722964+0000","last_undegraded":"2024-04-26T02:28:35.722964+0000","last_fullsized":"2024-04-26T02:28:35.722964+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6104,"log_dups_size":0,"ondisk_log_size":6104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:40.010851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46313996,"num_objects":5654,"num_object_clones":0,"num_object_copies":22616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6102,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6210","reported_seq":6288,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:35.582891+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:35.582891+0000","last_peered":"2024-04-26T02:28:35.582891+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:35.582891+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47337996,"num_objects":5779,"num_object_clones":0,"num_object_copies":23116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6022","reported_seq":6088,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.354061+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:32.354061+0000","last_peered":"2024-04-26T02:28:32.354061+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:32.354061+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46834712,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'5968","reported_seq":6064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:33.973777+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:33.973777+0000","last_peered":"2024-04-26T02:28:33.973777+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:33.973777+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45838860,"num_objects":5596,"num_object_clones":0,"num_object_copies":22384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3567}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6009","reported_seq":6070,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.116832+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:32.116832+0000","last_peered":"2024-04-26T02:28:32.116832+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:32.116832+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46770962,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'5955","reported_seq":6006,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.437469+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:32.437469+0000","last_peered":"2024-04-26T02:28:32.437469+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:32.437469+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217478,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6086","reported_seq":6181,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.057266+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:34.057266+0000","last_peered":"2024-04-26T02:28:34.057266+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:34.057266+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46635270,"num_objects":5693,"num_object_clones":0,"num_object_copies":22772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6226","reported_seq":6289,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:32.311610+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:32.311610+0000","last_peered":"2024-04-26T02:28:32.311610+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:32.311610+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48296460,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6137","reported_seq":9466,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:34.086814+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:34.086814+0000","last_peered":"2024-04-26T02:28:34.086814+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:34.086814+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46838284,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13295616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6128","reported_seq":6205,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:35.714985+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:35.714985+0000","last_peered":"2024-04-26T02:28:35.714985+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:35.714985+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46436876,"num_objects":5669,"num_object_clones":0,"num_object_copies":22676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":743447744,"num_objects":90760,"num_object_clones":0,"num_object_copies":363040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96797,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3814,"num_bytes_recovered":30351360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1378009088,"data_stored":1378009088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96824,"ondisk_log_size":96824,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729644,"kb_used_data":364768,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93642196,"statfs":{"total":96636764160,"available":95889608704,"internally_reserved":0,"allocated":373522432,"data_stored":373337310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,3,7,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727644,"kb_used_data":363280,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93644196,"statfs":{"total":96636764160,"available":95891656704,"internally_reserved":0,"allocated":371998720,"data_stored":371813598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,5,1,3,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510136,"kb_used_data":257776,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93861704,"statfs":{"total":96636764160,"available":96114384896,"internally_reserved":0,"allocated":263962624,"data_stored":263777502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258404047},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726964,"kb_used_data":362156,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93644876,"statfs":{"total":96636764160,"available":95892353024,"internally_reserved":0,"allocated":370847744,"data_stored":370662622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,0,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":369827840,"data_stored":369827840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263643136,"data_stored":263643136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371466240,"data_stored":371466240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373071872,"data_stored":373071872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:41.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:42.660 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:42.661 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:42.751 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-04-26T02:28:42.542389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733256896,"num_objects":89516,"num_object_clones":0,"num_object_copies":358064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98041,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":31047680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98068,"ondisk_log_size":98068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656144,"kb_used_data":1333292,"kb_used_omap":14,"kb_used_meta":1322801,"kb_avail":374831216,"statfs":{"total":386547056640,"available":383827165184,"internally_reserved":0,"allocated":1365291008,"data_stored":1364550520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1354548787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,10,15,15,12,6,0,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23863296,"num_objects":-2913,"num_object_clones":0,"num_object_copies":-11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.f","version":"34'6002","reported_seq":6064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.537978+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:37.537978+0000","last_peered":"2024-04-26T02:28:37.537978+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:37.537978+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45236224,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6231","reported_seq":6305,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.639425+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:40.639425+0000","last_peered":"2024-04-26T02:28:40.639425+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:40.639425+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45879820,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'6059","reported_seq":6155,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:38.922542+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:38.922542+0000","last_peered":"2024-04-26T02:28:38.922542+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:38.922542+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45635870,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6058,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6033","reported_seq":6109,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.656851+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:40.656851+0000","last_peered":"2024-04-26T02:28:40.656851+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:40.656851+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44618252,"num_objects":5447,"num_object_clones":0,"num_object_copies":21788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'6031","reported_seq":6103,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.540754+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:37.540754+0000","last_peered":"2024-04-26T02:28:37.540754+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:37.540754+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6031,"log_dups_size":0,"ondisk_log_size":6031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45593100,"num_objects":5566,"num_object_clones":0,"num_object_copies":22264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'6072","reported_seq":6128,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.545885+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:37.545885+0000","last_peered":"2024-04-26T02:28:37.545885+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:37.545885+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6185","reported_seq":10428,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:40.741644+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:40.741644+0000","last_peered":"2024-04-26T02:28:40.741644+0000","last_clean":"2024-04-26T02:28:40.741644+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:40.741644+0000","last_undegraded":"2024-04-26T02:28:40.741644+0000","last_fullsized":"2024-04-26T02:28:40.741644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45650444,"num_objects":5573,"num_object_clones":0,"num_object_copies":22292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6284","reported_seq":6367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.835553+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:40.835553+0000","last_peered":"2024-04-26T02:28:40.835553+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:40.835553+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46731788,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6095","reported_seq":6165,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.502146+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:37.502146+0000","last_peered":"2024-04-26T02:28:37.502146+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:37.502146+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46236696,"num_objects":5645,"num_object_clones":0,"num_object_copies":22580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'6040","reported_seq":6140,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:38.993767+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:38.993767+0000","last_peered":"2024-04-26T02:28:38.993767+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:38.993767+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249036,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3497}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6089","reported_seq":6154,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.622877+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:37.622877+0000","last_peered":"2024-04-26T02:28:37.622877+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:37.622877+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46115602,"num_objects":5630,"num_object_clones":0,"num_object_copies":22520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'6041","reported_seq":6107,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.604074+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:37.604074+0000","last_peered":"2024-04-26T02:28:37.604074+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:37.604074+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45512966,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6174","reported_seq":6281,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:39.040395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:39.040395+0000","last_peered":"2024-04-26T02:28:39.040395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:39.040395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914374,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6173,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6300","reported_seq":6367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.334450+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:37.334450+0000","last_peered":"2024-04-26T02:28:37.334450+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:37.334450+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6300,"log_dups_size":0,"ondisk_log_size":6300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47690252,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6228","reported_seq":9727,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:39.107696+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:39.107696+0000","last_peered":"2024-04-26T02:28:39.107696+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:39.107696+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46092812,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6204","reported_seq":6284,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.711246+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:40.711246+0000","last_peered":"2024-04-26T02:28:40.711246+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:40.711246+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45814284,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733256896,"num_objects":89516,"num_object_clones":0,"num_object_copies":358064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98041,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":31047680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1362530304,"data_stored":1362530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98068,"ondisk_log_size":98068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716572,"kb_used_data":360020,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93655268,"statfs":{"total":96636764160,"available":95902994432,"internally_reserved":0,"allocated":368660480,"data_stored":368475358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,4,4,4,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710392,"kb_used_data":358508,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93661448,"statfs":{"total":96636764160,"available":95909322752,"internally_reserved":0,"allocated":367112192,"data_stored":366927070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,8,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519576,"kb_used_data":257488,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93852264,"statfs":{"total":96636764160,"available":96104718336,"internally_reserved":0,"allocated":263667712,"data_stored":263482590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709604,"kb_used_data":357276,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93662236,"statfs":{"total":96636764160,"available":95910129664,"internally_reserved":0,"allocated":365850624,"data_stored":365665502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,3,3,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364666880,"data_stored":364666880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263331840,"data_stored":263331840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366534656,"data_stored":366534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367996928,"data_stored":367996928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:42.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:43.830 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:43.877 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-04-26T02:28:42.542389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":733256896,"num_objects":89516,"num_object_clones":0,"num_object_copies":358064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98041,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":31047680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98068,"ondisk_log_size":98068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656144,"kb_used_data":1333292,"kb_used_omap":14,"kb_used_meta":1322801,"kb_avail":374831216,"statfs":{"total":386547056640,"available":383827165184,"internally_reserved":0,"allocated":1365291008,"data_stored":1364550520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1354548787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,10,15,15,12,6,0,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23863296,"num_objects":-2913,"num_object_clones":0,"num_object_copies":-11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.f","version":"34'6002","reported_seq":6064,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.537978+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:37.537978+0000","last_peered":"2024-04-26T02:28:37.537978+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:37.537978+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45236224,"num_objects":5522,"num_object_clones":0,"num_object_copies":22088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6231","reported_seq":6305,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.639425+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:40.639425+0000","last_peered":"2024-04-26T02:28:40.639425+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:40.639425+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45879820,"num_objects":5601,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3529}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'6059","reported_seq":6155,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:38.922542+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:38.922542+0000","last_peered":"2024-04-26T02:28:38.922542+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:38.922542+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45635870,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6058,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6033","reported_seq":6109,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.656851+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:40.656851+0000","last_peered":"2024-04-26T02:28:40.656851+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:40.656851+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44618252,"num_objects":5447,"num_object_clones":0,"num_object_copies":21788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'6031","reported_seq":6103,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.540754+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:37.540754+0000","last_peered":"2024-04-26T02:28:37.540754+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:37.540754+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6031,"log_dups_size":0,"ondisk_log_size":6031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45593100,"num_objects":5566,"num_object_clones":0,"num_object_copies":22264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'6072","reported_seq":6128,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.545885+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:37.545885+0000","last_peered":"2024-04-26T02:28:37.545885+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:37.545885+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6185","reported_seq":10428,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:40.741644+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:40.741644+0000","last_peered":"2024-04-26T02:28:40.741644+0000","last_clean":"2024-04-26T02:28:40.741644+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:40.741644+0000","last_undegraded":"2024-04-26T02:28:40.741644+0000","last_fullsized":"2024-04-26T02:28:40.741644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45650444,"num_objects":5573,"num_object_clones":0,"num_object_copies":22292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6284","reported_seq":6367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.835553+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:40.835553+0000","last_peered":"2024-04-26T02:28:40.835553+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:40.835553+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46731788,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6095","reported_seq":6165,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.502146+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:37.502146+0000","last_peered":"2024-04-26T02:28:37.502146+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:37.502146+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46236696,"num_objects":5645,"num_object_clones":0,"num_object_copies":22580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'6040","reported_seq":6140,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:38.993767+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:38.993767+0000","last_peered":"2024-04-26T02:28:38.993767+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:38.993767+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249036,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3497}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6089","reported_seq":6154,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.622877+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:37.622877+0000","last_peered":"2024-04-26T02:28:37.622877+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:37.622877+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46115602,"num_objects":5630,"num_object_clones":0,"num_object_copies":22520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'6041","reported_seq":6107,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.604074+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:37.604074+0000","last_peered":"2024-04-26T02:28:37.604074+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:37.604074+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45512966,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6174","reported_seq":6281,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:39.040395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:39.040395+0000","last_peered":"2024-04-26T02:28:39.040395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:39.040395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914374,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6173,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6300","reported_seq":6367,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:37.334450+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:37.334450+0000","last_peered":"2024-04-26T02:28:37.334450+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:37.334450+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6300,"log_dups_size":0,"ondisk_log_size":6300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47690252,"num_objects":5822,"num_object_clones":0,"num_object_copies":23288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6228","reported_seq":9727,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:39.107696+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:39.107696+0000","last_peered":"2024-04-26T02:28:39.107696+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:39.107696+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46092812,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13991936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6204","reported_seq":6284,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:40.711246+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:40.711246+0000","last_peered":"2024-04-26T02:28:40.711246+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:40.711246+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45814284,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":733256896,"num_objects":89516,"num_object_clones":0,"num_object_copies":358064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98041,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":31047680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1362530304,"data_stored":1362530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98068,"ondisk_log_size":98068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716572,"kb_used_data":360020,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93655268,"statfs":{"total":96636764160,"available":95902994432,"internally_reserved":0,"allocated":368660480,"data_stored":368475358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,4,4,4,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710392,"kb_used_data":358508,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93661448,"statfs":{"total":96636764160,"available":95909322752,"internally_reserved":0,"allocated":367112192,"data_stored":366927070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,8,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519576,"kb_used_data":257488,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93852264,"statfs":{"total":96636764160,"available":96104718336,"internally_reserved":0,"allocated":263667712,"data_stored":263482590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709604,"kb_used_data":357276,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93662236,"statfs":{"total":96636764160,"available":95910129664,"internally_reserved":0,"allocated":365850624,"data_stored":365665502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,3,3,3,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364666880,"data_stored":364666880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263331840,"data_stored":263331840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366534656,"data_stored":366534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367996928,"data_stored":367996928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'6002", 'reported_seq': 6064, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.537978+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:37.537978+0000', 'last_peered': '2024-04-26T02:28:37.537978+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:37.537978+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6002, 'log_dups_size': 0, 'ondisk_log_size': 6002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45236224, 'num_objects': 5522, 'num_object_clones': 0, 'num_object_copies': 22088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6002, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'6231", 'reported_seq': 6305, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:40.639425+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:40.639425+0000', 'last_peered': '2024-04-26T02:28:40.639425+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:40.639425+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6231, 'log_dups_size': 0, 'ondisk_log_size': 6231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45879820, 'num_objects': 5601, 'num_object_clones': 0, 'num_object_copies': 22404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2074, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6229, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'6059", 'reported_seq': 6155, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:38.922542+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:38.922542+0000', 'last_peered': '2024-04-26T02:28:38.922542+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:38.922542+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6059, 'log_dups_size': 0, 'ondisk_log_size': 6059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45635870, 'num_objects': 5572, 'num_object_clones': 0, 'num_object_copies': 22288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5572, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6058, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3556}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'6033", 'reported_seq': 6109, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:40.656851+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:40.656851+0000', 'last_peered': '2024-04-26T02:28:40.656851+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:40.656851+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6033, 'log_dups_size': 0, 'ondisk_log_size': 6033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44618252, 'num_objects': 5447, 'num_object_clones': 0, 'num_object_copies': 21788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6031, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'6031", 'reported_seq': 6103, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.540754+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:37.540754+0000', 'last_peered': '2024-04-26T02:28:37.540754+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:37.540754+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6031, 'log_dups_size': 0, 'ondisk_log_size': 6031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45593100, 'num_objects': 5566, 'num_object_clones': 0, 'num_object_copies': 22264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6029, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3563}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:43.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'6072", 'reported_seq': 6128, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.545885+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:37.545885+0000', 'last_peered': '2024-04-26T02:28:37.545885+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:37.545885+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6072, 'log_dups_size': 0, 'ondisk_log_size': 6072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45285376, 'num_objects': 5528, 'num_object_clones': 0, 'num_object_copies': 22112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6072, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'6284", 'reported_seq': 6367, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:40.835553+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:40.835553+0000', 'last_peered': '2024-04-26T02:28:40.835553+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:40.835553+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6284, 'log_dups_size': 0, 'ondisk_log_size': 6284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46731788, 'num_objects': 5705, 'num_object_clones': 0, 'num_object_copies': 22820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2135, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6282, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'6095", 'reported_seq': 6165, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.502146+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:37.502146+0000', 'last_peered': '2024-04-26T02:28:37.502146+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:37.502146+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6095, 'log_dups_size': 0, 'ondisk_log_size': 6095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46236696, 'num_objects': 5645, 'num_object_clones': 0, 'num_object_copies': 22580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2001, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6092, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'6040", 'reported_seq': 6140, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:38.993767+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:38.993767+0000', 'last_peered': '2024-04-26T02:28:38.993767+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:38.993767+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6040, 'log_dups_size': 0, 'ondisk_log_size': 6040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45249036, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6038, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3497}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'6089", 'reported_seq': 6154, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.622877+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:37.622877+0000', 'last_peered': '2024-04-26T02:28:37.622877+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:37.622877+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6089, 'log_dups_size': 0, 'ondisk_log_size': 6089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46115602, 'num_objects': 5630, 'num_object_clones': 0, 'num_object_copies': 22520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6086, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'6041", 'reported_seq': 6107, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.604074+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:37.604074+0000', 'last_peered': '2024-04-26T02:28:37.604074+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:37.604074+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6041, 'log_dups_size': 0, 'ondisk_log_size': 6041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45512966, 'num_objects': 5556, 'num_object_clones': 0, 'num_object_copies': 22224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6040, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'6174", 'reported_seq': 6281, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:39.040395+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:39.040395+0000', 'last_peered': '2024-04-26T02:28:39.040395+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:39.040395+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6174, 'log_dups_size': 0, 'ondisk_log_size': 6174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45914374, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6173, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'6300", 'reported_seq': 6367, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:37.334450+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:37.334450+0000', 'last_peered': '2024-04-26T02:28:37.334450+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:37.334450+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6300, 'log_dups_size': 0, 'ondisk_log_size': 6300, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47690252, 'num_objects': 5822, 'num_object_clones': 0, 'num_object_copies': 23288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6298, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'6228", 'reported_seq': 9727, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:39.107696+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:39.107696+0000', 'last_peered': '2024-04-26T02:28:39.107696+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:39.107696+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6228, 'log_dups_size': 0, 'ondisk_log_size': 6228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46092812, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 22508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 345, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6226, 'num_write_kb': 47413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1712, 'num_bytes_recovered': 13991936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 5282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:43.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'6204", 'reported_seq': 6284, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:40.711246+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:40.711246+0000', 'last_peered': '2024-04-26T02:28:40.711246+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:40.711246+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6204, 'log_dups_size': 0, 'ondisk_log_size': 6204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45814284, 'num_objects': 5593, 'num_object_clones': 0, 'num_object_copies': 22372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6202, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:43.882 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:45.401 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:45.401 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28688/354612 objects degraded (8.090%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":321,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88653,"data_bytes":726187200,"bytes_used":2726436864,"bytes_avail":383820619776,"bytes_total":386547056640,"degraded_objects":28688,"degraded_total":354612,"degraded_ratio":0.080899687545824731,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":199970,"recovering_keys_per_sec":0,"num_objects_recovered":373,"num_bytes_recovered":2400256,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":236},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:45.433 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:45.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:46.771 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:46.772 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:46.901 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-04-26T02:28:46.543568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":723238080,"num_objects":88293,"num_object_clones":0,"num_object_copies":353172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99264,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4094,"num_bytes_recovered":31973376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99291,"ondisk_log_size":99291,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666084,"kb_used_data":1318252,"kb_used_omap":14,"kb_used_meta":1347761,"kb_avail":374821276,"statfs":{"total":386547056640,"available":383816986624,"internally_reserved":0,"allocated":1349890048,"data_stored":1349149560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1380107827},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,11,8,9,6,7,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23502848,"num_objects":-2869,"num_object_clones":0,"num_object_copies":-11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003109"},"pg_stats":[{"pgid":"1.f","version":"34'6079","reported_seq":6143,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.651533+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:42.651533+0000","last_peered":"2024-04-26T02:28:42.651533+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:42.651533+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6298","reported_seq":6375,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.898103+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:45.898103+0000","last_peered":"2024-04-26T02:28:45.898103+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:45.898103+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45330956,"num_objects":5534,"num_object_clones":0,"num_object_copies":22136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'6140","reported_seq":6238,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:44.112509+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:44.112509+0000","last_peered":"2024-04-26T02:28:44.112509+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:44.112509+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44972318,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6139,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6105","reported_seq":6186,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.724500+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:45.724500+0000","last_peered":"2024-04-26T02:28:45.724500+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:45.724500+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44028428,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6103,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'6115","reported_seq":6208,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.589266+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:42.589266+0000","last_peered":"2024-04-26T02:28:42.589266+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:42.589266+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44904972,"num_objects":5482,"num_object_clones":0,"num_object_copies":21928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'6149","reported_seq":6225,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.667904+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:42.667904+0000","last_peered":"2024-04-26T02:28:42.667904+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:42.667904+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44654592,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6149,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6249","reported_seq":10492,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:45.747186+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:45.747186+0000","last_peered":"2024-04-26T02:28:45.747186+0000","last_clean":"2024-04-26T02:28:45.747186+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:45.747186+0000","last_undegraded":"2024-04-26T02:28:45.747186+0000","last_fullsized":"2024-04-26T02:28:45.747186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6249,"log_dups_size":0,"ondisk_log_size":6249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:45.951386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45126156,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6363","reported_seq":6460,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.835402+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:45.835402+0000","last_peered":"2024-04-26T02:28:45.835402+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:45.835402+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46084620,"num_objects":5626,"num_object_clones":0,"num_object_copies":22504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6169","reported_seq":6245,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.541612+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:42.541612+0000","last_peered":"2024-04-26T02:28:42.541612+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:42.541612+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6169,"log_dups_size":0,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45630488,"num_objects":5571,"num_object_clones":0,"num_object_copies":22284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'6118","reported_seq":6218,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:44.116177+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:44.116177+0000","last_peered":"2024-04-26T02:28:44.116177+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:44.116177+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6118,"log_dups_size":0,"ondisk_log_size":6118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44610060,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6182","reported_seq":6259,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.615033+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:42.615033+0000","last_peered":"2024-04-26T02:28:42.615033+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:42.615033+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45353746,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'6112","reported_seq":6182,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.704077+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:42.704077+0000","last_peered":"2024-04-26T02:28:42.704077+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:42.704077+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44931334,"num_objects":5485,"num_object_clones":0,"num_object_copies":21940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6248","reported_seq":6358,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:43.984674+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:43.984674+0000","last_peered":"2024-04-26T02:28:43.984674+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:43.984674+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45308166,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6383","reported_seq":6451,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:42.731563+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:42.731563+0000","last_peered":"2024-04-26T02:28:42.731563+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:42.731563+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47010316,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6299","reported_seq":10026,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:44.125793+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:44.125793+0000","last_peered":"2024-04-26T02:28:44.125793+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:44.125793+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45511180,"num_objects":5556,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6297,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1825,"num_bytes_recovered":14917632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6282","reported_seq":6368,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.832751+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:45.832751+0000","last_peered":"2024-04-26T02:28:45.832751+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:45.832751+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45175308,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":723238080,"num_objects":88293,"num_object_clones":0,"num_object_copies":353172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99264,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4094,"num_bytes_recovered":31973376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347006464,"data_stored":1347006464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99291,"ondisk_log_size":99291,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715772,"kb_used_data":355044,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93656068,"statfs":{"total":96636764160,"available":95903813632,"internally_reserved":0,"allocated":363565056,"data_stored":363379934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,7,1,4,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713744,"kb_used_data":353544,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93658096,"statfs":{"total":96636764160,"available":95905890304,"internally_reserved":0,"allocated":362029056,"data_stored":361843934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1,2,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523416,"kb_used_data":257168,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93848424,"statfs":{"total":96636764160,"available":96100786176,"internally_reserved":0,"allocated":263340032,"data_stored":263154910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713152,"kb_used_data":352496,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93658688,"statfs":{"total":96636764160,"available":95906496512,"internally_reserved":0,"allocated":360955904,"data_stored":360770782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359821312,"data_stored":359821312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263041024,"data_stored":263041024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361435136,"data_stored":361435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362708992,"data_stored":362708992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:49.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:50.798 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:50.854 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-04-26T02:28:50.547624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":716356800,"num_objects":87453,"num_object_clones":0,"num_object_copies":349812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100104,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4245,"num_bytes_recovered":33062912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100131,"ondisk_log_size":100131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673180,"kb_used_data":1308720,"kb_used_omap":14,"kb_used_meta":1364401,"kb_avail":374814180,"statfs":{"total":386547056640,"available":383809720320,"internally_reserved":0,"allocated":1340129280,"data_stored":1339388792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1397147187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,8,12,7,7,5,4,0,1,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22618112,"num_objects":-2761,"num_object_clones":0,"num_object_copies":-11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-413,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":2711552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006132"},"pg_stats":[{"pgid":"1.f","version":"34'6158","reported_seq":6231,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.743985+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:47.743985+0000","last_peered":"2024-04-26T02:28:47.743985+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:47.743985+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43958272,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6298","reported_seq":6375,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.898103+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:45.898103+0000","last_peered":"2024-04-26T02:28:45.898103+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:45.898103+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45330956,"num_objects":5534,"num_object_clones":0,"num_object_copies":22136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'6217","reported_seq":6324,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.183603+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:49.183603+0000","last_peered":"2024-04-26T02:28:49.183603+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:49.183603+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44341534,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6216,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3405}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6105","reported_seq":6186,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.724500+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:45.724500+0000","last_peered":"2024-04-26T02:28:45.724500+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:45.724500+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44028428,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6103,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'6185","reported_seq":6282,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.751527+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:47.751527+0000","last_peered":"2024-04-26T02:28:47.751527+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:47.751527+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44331532,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'6227","reported_seq":6305,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.798108+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:47.798108+0000","last_peered":"2024-04-26T02:28:47.798108+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:47.798108+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6227,"log_dups_size":0,"ondisk_log_size":6227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44015616,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6249","reported_seq":10492,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:45.747186+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:45.747186+0000","last_peered":"2024-04-26T02:28:45.747186+0000","last_clean":"2024-04-26T02:28:45.747186+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:45.747186+0000","last_undegraded":"2024-04-26T02:28:45.747186+0000","last_fullsized":"2024-04-26T02:28:45.747186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6249,"log_dups_size":0,"ondisk_log_size":6249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:45.951386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45126156,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6247,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6363","reported_seq":6460,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.835402+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:45.835402+0000","last_peered":"2024-04-26T02:28:45.835402+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:45.835402+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46084620,"num_objects":5626,"num_object_clones":0,"num_object_copies":22504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6242","reported_seq":6322,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.642196+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:47.642196+0000","last_peered":"2024-04-26T02:28:47.642196+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:47.642196+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45032472,"num_objects":5498,"num_object_clones":0,"num_object_copies":21992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'6202","reported_seq":6304,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.023420+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:49.023420+0000","last_peered":"2024-04-26T02:28:49.023420+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:49.023420+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43921932,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6200,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6251","reported_seq":6328,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.820318+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:47.820318+0000","last_peered":"2024-04-26T02:28:47.820318+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:47.820318+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44788498,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'6194","reported_seq":6266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.750414+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:47.750414+0000","last_peered":"2024-04-26T02:28:47.750414+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:47.750414+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44259590,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6193,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6321","reported_seq":6431,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.150663+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:49.150663+0000","last_peered":"2024-04-26T02:28:49.150663+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:49.150663+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44710150,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6465","reported_seq":6535,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.623227+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:47.623227+0000","last_peered":"2024-04-26T02:28:47.623227+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:47.623227+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46338572,"num_objects":5657,"num_object_clones":0,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3594}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6372","reported_seq":10365,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.185973+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:49.185973+0000","last_peered":"2024-04-26T02:28:49.185973+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:49.185973+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913164,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6370,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1958,"num_bytes_recovered":16007168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6282","reported_seq":6368,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:45.832751+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:45.832751+0000","last_peered":"2024-04-26T02:28:45.832751+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:45.832751+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45175308,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":716356800,"num_objects":87453,"num_object_clones":0,"num_object_copies":349812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100104,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4245,"num_bytes_recovered":33062912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337008128,"data_stored":1337008128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100131,"ondisk_log_size":100131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715192,"kb_used_data":350320,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93656648,"statfs":{"total":96636764160,"available":95904407552,"internally_reserved":0,"allocated":358727680,"data_stored":358542558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,2,5,2,4,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713136,"kb_used_data":348772,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93658704,"statfs":{"total":96636764160,"available":95906512896,"internally_reserved":0,"allocated":357142528,"data_stored":356957406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531700,"kb_used_data":257132,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93840140,"statfs":{"total":96636764160,"available":96092303360,"internally_reserved":0,"allocated":263303168,"data_stored":263118046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713152,"kb_used_data":352496,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93658688,"statfs":{"total":96636764160,"available":95906496512,"internally_reserved":0,"allocated":360955904,"data_stored":360770782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359821312,"data_stored":359821312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263020544,"data_stored":263020544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356372480,"data_stored":356372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357793792,"data_stored":357793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:50.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:52.399 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:52.400 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:52.504 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-04-26T02:28:51.760940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713260224,"num_objects":87075,"num_object_clones":0,"num_object_copies":348300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100482,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33062912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100509,"ondisk_log_size":100509,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672364,"kb_used_data":1303752,"kb_used_omap":14,"kb_used_meta":1368561,"kb_avail":374814996,"statfs":{"total":386547056640,"available":383810555904,"internally_reserved":0,"allocated":1335042048,"data_stored":1334301560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1401407027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,10,10,6,8,5,4,2,1,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23134208,"num_objects":-2824,"num_object_clones":0,"num_object_copies":-11296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":2015232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.218951"},"pg_stats":[{"pgid":"1.f","version":"34'6158","reported_seq":6231,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.743985+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:47.743985+0000","last_peered":"2024-04-26T02:28:47.743985+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:47.743985+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43958272,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6358","reported_seq":6439,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.792977+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:50.792977+0000","last_peered":"2024-04-26T02:28:50.792977+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:50.792977+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44839436,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"34'6217","reported_seq":6324,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.183603+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:49.183603+0000","last_peered":"2024-04-26T02:28:49.183603+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:49.183603+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6217,"log_dups_size":0,"ondisk_log_size":6217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44341534,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6216,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3405}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6177","reported_seq":6263,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.885826+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:50.885826+0000","last_peered":"2024-04-26T02:28:50.885826+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:50.885826+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43438604,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6175,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'6185","reported_seq":6282,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.751527+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:47.751527+0000","last_peered":"2024-04-26T02:28:47.751527+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:47.751527+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44331532,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'6227","reported_seq":6305,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.798108+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:47.798108+0000","last_peered":"2024-04-26T02:28:47.798108+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:47.798108+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6227,"log_dups_size":0,"ondisk_log_size":6227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44015616,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6333","reported_seq":10576,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:50.787893+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:50.787893+0000","last_peered":"2024-04-26T02:28:50.787893+0000","last_clean":"2024-04-26T02:28:50.787893+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:50.787893+0000","last_undegraded":"2024-04-26T02:28:50.787893+0000","last_fullsized":"2024-04-26T02:28:50.787893+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:51.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44438028,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6446","reported_seq":6551,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.665165+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:50.665165+0000","last_peered":"2024-04-26T02:28:50.665165+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:50.665165+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45404684,"num_objects":5543,"num_object_clones":0,"num_object_copies":22172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6444,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'6242","reported_seq":6322,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.642196+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:47.642196+0000","last_peered":"2024-04-26T02:28:47.642196+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:47.642196+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45032472,"num_objects":5498,"num_object_clones":0,"num_object_copies":21992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3505}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'6202","reported_seq":6304,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.023420+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:49.023420+0000","last_peered":"2024-04-26T02:28:49.023420+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:49.023420+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43921932,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6200,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'6251","reported_seq":6328,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.820318+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:47.820318+0000","last_peered":"2024-04-26T02:28:47.820318+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:47.820318+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44788498,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'6194","reported_seq":6266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.750414+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:47.750414+0000","last_peered":"2024-04-26T02:28:47.750414+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:47.750414+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44259590,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6193,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'6321","reported_seq":6431,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.150663+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:49.150663+0000","last_peered":"2024-04-26T02:28:49.150663+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:49.150663+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44710150,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'6465","reported_seq":6535,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:47.623227+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:47.623227+0000","last_peered":"2024-04-26T02:28:47.623227+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:47.623227+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46338572,"num_objects":5657,"num_object_clones":0,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3594}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"34'6372","reported_seq":10365,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:49.185973+0000","last_change":"2024-04-26T02:26:44.670105+0000","last_active":"2024-04-26T02:28:49.185973+0000","last_peered":"2024-04-26T02:28:49.185973+0000","last_clean":"2024-04-26T02:23:51.681362+0000","last_became_active":"2024-04-26T02:24:59.509785+0000","last_became_peered":"2024-04-26T02:24:59.509785+0000","last_unstale":"2024-04-26T02:28:49.185973+0000","last_undegraded":"2024-04-26T02:24:59.291954+0000","last_fullsized":"2024-04-26T02:24:59.229944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44913164,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6370,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1958,"num_bytes_recovered":16007168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":5384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'6361","reported_seq":6452,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.830908+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:50.830908+0000","last_peered":"2024-04-26T02:28:50.830908+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:50.830908+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6361,"log_dups_size":0,"ondisk_log_size":6361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44528140,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":713260224,"num_objects":87075,"num_object_clones":0,"num_object_copies":348300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100482,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4256,"num_bytes_recovered":33062912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1331998720,"data_stored":1331998720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100509,"ondisk_log_size":100509,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715192,"kb_used_data":350320,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93656648,"statfs":{"total":96636764160,"available":95904407552,"internally_reserved":0,"allocated":358727680,"data_stored":358542558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,2,5,2,4,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713136,"kb_used_data":348772,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93658704,"statfs":{"total":96636764160,"available":95906512896,"internally_reserved":0,"allocated":357142528,"data_stored":356957406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531700,"kb_used_data":257132,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93840140,"statfs":{"total":96636764160,"available":96092303360,"internally_reserved":0,"allocated":263303168,"data_stored":263118046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712336,"kb_used_data":347528,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93659504,"statfs":{"total":96636764160,"available":95907332096,"internally_reserved":0,"allocated":355868672,"data_stored":355683550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354811904,"data_stored":354811904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263020544,"data_stored":263020544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356372480,"data_stored":356372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357793792,"data_stored":357793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'6158", 'reported_seq': 6231, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.743985+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:47.743985+0000', 'last_peered': '2024-04-26T02:28:47.743985+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:47.743985+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6158, 'log_dups_size': 0, 'ondisk_log_size': 6158, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43958272, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6158, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3364}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'6358", 'reported_seq': 6439, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:50.792977+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:50.792977+0000', 'last_peered': '2024-04-26T02:28:50.792977+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:50.792977+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6358, 'log_dups_size': 0, 'ondisk_log_size': 6358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44839436, 'num_objects': 5474, 'num_object_clones': 0, 'num_object_copies': 21896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6356, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "34'6217", 'reported_seq': 6324, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:49.183603+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:49.183603+0000', 'last_peered': '2024-04-26T02:28:49.183603+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:49.183603+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6217, 'log_dups_size': 0, 'ondisk_log_size': 6217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44341534, 'num_objects': 5414, 'num_object_clones': 0, 'num_object_copies': 21656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2010, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5414, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6216, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3405}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'6177", 'reported_seq': 6263, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:50.885826+0000', 'last_change': '2024-04-26T02:24:59.560749+0000', 'last_active': '2024-04-26T02:28:50.885826+0000', 'last_peered': '2024-04-26T02:28:50.885826+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:50.885826+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6177, 'log_dups_size': 0, 'ondisk_log_size': 6177, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43438604, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6175, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:28:52.505 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'6185", 'reported_seq': 6282, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.751527+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:47.751527+0000', 'last_peered': '2024-04-26T02:28:47.751527+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:47.751527+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6185, 'log_dups_size': 0, 'ondisk_log_size': 6185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44331532, 'num_objects': 5412, 'num_object_clones': 0, 'num_object_copies': 21648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6183, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'6227", 'reported_seq': 6305, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.798108+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:47.798108+0000', 'last_peered': '2024-04-26T02:28:47.798108+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:47.798108+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6227, 'log_dups_size': 0, 'ondisk_log_size': 6227, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44015616, 'num_objects': 5373, 'num_object_clones': 0, 'num_object_copies': 21492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6227, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'6446", 'reported_seq': 6551, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:50.665165+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:50.665165+0000', 'last_peered': '2024-04-26T02:28:50.665165+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:50.665165+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6446, 'log_dups_size': 0, 'ondisk_log_size': 6446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45404684, 'num_objects': 5543, 'num_object_clones': 0, 'num_object_copies': 22172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2124, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6444, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "34'6242", 'reported_seq': 6322, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.642196+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:47.642196+0000', 'last_peered': '2024-04-26T02:28:47.642196+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:47.642196+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6242, 'log_dups_size': 0, 'ondisk_log_size': 6242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45032472, 'num_objects': 5498, 'num_object_clones': 0, 'num_object_copies': 21992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1996, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6239, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3505}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "34'6202", 'reported_seq': 6304, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:49.023420+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:49.023420+0000', 'last_peered': '2024-04-26T02:28:49.023420+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:49.023420+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6202, 'log_dups_size': 0, 'ondisk_log_size': 6202, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43921932, 'num_objects': 5362, 'num_object_clones': 0, 'num_object_copies': 21448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6200, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:28:52.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'6251", 'reported_seq': 6328, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.820318+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:47.820318+0000', 'last_peered': '2024-04-26T02:28:47.820318+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:47.820318+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6251, 'log_dups_size': 0, 'ondisk_log_size': 6251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44788498, 'num_objects': 5468, 'num_object_clones': 0, 'num_object_copies': 21872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2011, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6248, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3460}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'6194", 'reported_seq': 6266, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.750414+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:47.750414+0000', 'last_peered': '2024-04-26T02:28:47.750414+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:47.750414+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6194, 'log_dups_size': 0, 'ondisk_log_size': 6194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44259590, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 21612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6193, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "34'6321", 'reported_seq': 6431, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:49.150663+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:49.150663+0000', 'last_peered': '2024-04-26T02:28:49.150663+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:49.150663+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6321, 'log_dups_size': 0, 'ondisk_log_size': 6321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44710150, 'num_objects': 5458, 'num_object_clones': 0, 'num_object_copies': 21832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6320, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'6465", 'reported_seq': 6535, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:47.623227+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:47.623227+0000', 'last_peered': '2024-04-26T02:28:47.623227+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:47.623227+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6465, 'log_dups_size': 0, 'ondisk_log_size': 6465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46338572, 'num_objects': 5657, 'num_object_clones': 0, 'num_object_copies': 22628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6463, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "34'6372", 'reported_seq': 10365, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:49.185973+0000', 'last_change': '2024-04-26T02:26:44.670105+0000', 'last_active': '2024-04-26T02:28:49.185973+0000', 'last_peered': '2024-04-26T02:28:49.185973+0000', 'last_clean': '2024-04-26T02:23:51.681362+0000', 'last_became_active': '2024-04-26T02:24:59.509785+0000', 'last_became_peered': '2024-04-26T02:24:59.509785+0000', 'last_unstale': '2024-04-26T02:28:49.185973+0000', 'last_undegraded': '2024-04-26T02:24:59.291954+0000', 'last_fullsized': '2024-04-26T02:24:59.229944+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6372, 'log_dups_size': 0, 'ondisk_log_size': 6372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44913164, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 99, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6370, 'num_write_kb': 47413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1958, 'num_bytes_recovered': 16007168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 5384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:28:52.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'6361", 'reported_seq': 6452, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:50.830908+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:50.830908+0000', 'last_peered': '2024-04-26T02:28:50.830908+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:50.830908+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6361, 'log_dups_size': 0, 'ondisk_log_size': 6361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44528140, 'num_objects': 5436, 'num_object_clones': 0, 'num_object_copies': 21744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6359, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3385}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:28:52.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:28:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:53.883 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28513/348300 objects degraded (8.186%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":330,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87075,"data_bytes":713260224,"bytes_used":2736500736,"bytes_avail":383810555904,"bytes_total":386547056640,"degraded_objects":28513,"degraded_total":348300,"degraded_ratio":0.081863336204421475,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":201388,"recovering_keys_per_sec":0,"num_objects_recovered":322,"num_bytes_recovered":2015232,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":243},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:28:53.965 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:28:53.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:55.267 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:55.268 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:55.350 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-04-26T02:28:54.550545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":706821312,"num_objects":86289,"num_object_clones":0,"num_object_copies":345156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101268,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4389,"num_bytes_recovered":33873920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101295,"ondisk_log_size":101295,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630004,"kb_used_data":1294664,"kb_used_omap":14,"kb_used_meta":1335281,"kb_avail":374857356,"statfs":{"total":386547056640,"available":383853932544,"internally_reserved":0,"allocated":1325735936,"data_stored":1324977003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1367328307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,9,5,5,5,6,3,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16416768,"num_objects":-2004,"num_object_clones":0,"num_object_copies":-8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006977"},"pg_stats":[{"pgid":"1.f","version":"36'6242","reported_seq":6327,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.921274+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:52.921274+0000","last_peered":"2024-04-26T02:28:52.921274+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:52.921274+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43270144,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'6358","reported_seq":6439,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.792977+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:50.792977+0000","last_peered":"2024-04-26T02:28:50.792977+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:50.792977+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44839436,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6293","reported_seq":6411,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.167721+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:54.167721+0000","last_peered":"2024-04-26T02:28:54.167721+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:54.167721+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43718942,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6292,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'6177","reported_seq":6263,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.885826+0000","last_change":"2024-04-26T02:24:59.560749+0000","last_active":"2024-04-26T02:28:50.885826+0000","last_peered":"2024-04-26T02:28:50.885826+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:50.885826+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43438604,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6175,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6260","reported_seq":6369,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.844143+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:52.844143+0000","last_peered":"2024-04-26T02:28:52.844143+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:52.844143+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717132,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6303","reported_seq":6394,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.929361+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:52.929361+0000","last_peered":"2024-04-26T02:28:52.929361+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:52.929361+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43393024,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'6333","reported_seq":10576,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-26T02:28:50.787893+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:50.787893+0000","last_peered":"2024-04-26T02:28:50.787893+0000","last_clean":"2024-04-26T02:28:50.787893+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:50.787893+0000","last_undegraded":"2024-04-26T02:28:50.787893+0000","last_fullsized":"2024-04-26T02:28:50.787893+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:51.000250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44438028,"num_objects":5425,"num_object_clones":0,"num_object_copies":21700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'6446","reported_seq":6551,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.665165+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:50.665165+0000","last_peered":"2024-04-26T02:28:50.665165+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:50.665165+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45404684,"num_objects":5543,"num_object_clones":0,"num_object_copies":22172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6444,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6317","reported_seq":6415,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.813522+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:52.813522+0000","last_peered":"2024-04-26T02:28:52.813522+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:52.813522+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44418072,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6261","reported_seq":6367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.159394+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:54.159394+0000","last_peered":"2024-04-26T02:28:54.159394+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:54.159394+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43438604,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6330","reported_seq":6415,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.907376+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:52.907376+0000","last_peered":"2024-04-26T02:28:52.907376+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:52.907376+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44141330,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6327,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3383}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6275","reported_seq":6363,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.953278+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:52.953278+0000","last_peered":"2024-04-26T02:28:52.953278+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:52.953278+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596038,"num_objects":5322,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6398","reported_seq":6514,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.074514+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:54.074514+0000","last_peered":"2024-04-26T02:28:54.074514+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:54.074514+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079366,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6532","reported_seq":6614,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:52.812355+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:52.812355+0000","last_peered":"2024-04-26T02:28:52.812355+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:52.812355+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45789708,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3531}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6409","reported_seq":10609,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:52.824678+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:28:52.824678+0000","last_peered":"2024-04-26T02:28:52.824678+0000","last_clean":"2024-04-26T02:28:52.824678+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:28:52.824678+0000","last_undegraded":"2024-04-26T02:28:52.824678+0000","last_fullsized":"2024-04-26T02:28:52.824678+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44610060,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"34'6361","reported_seq":6452,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:50.830908+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:50.830908+0000","last_peered":"2024-04-26T02:28:50.830908+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:50.830908+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6361,"log_dups_size":0,"ondisk_log_size":6361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44528140,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":706821312,"num_objects":86289,"num_object_clones":0,"num_object_copies":345156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101268,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4389,"num_bytes_recovered":33873920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1322541056,"data_stored":1322541056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101295,"ondisk_log_size":101295,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702160,"kb_used_data":345604,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93669680,"statfs":{"total":96636764160,"available":95917752320,"internally_reserved":0,"allocated":353898496,"data_stored":353708768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,2,1,3,6,1,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696168,"kb_used_data":344284,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93675672,"statfs":{"total":96636764160,"available":95923888128,"internally_reserved":0,"allocated":352546816,"data_stored":352352461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519340,"kb_used_data":257248,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93852500,"statfs":{"total":96636764160,"available":96104960000,"internally_reserved":0,"allocated":263421952,"data_stored":263232224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712336,"kb_used_data":347528,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93659504,"statfs":{"total":96636764160,"available":95907332096,"internally_reserved":0,"allocated":355868672,"data_stored":355683550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354811904,"data_stored":354811904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263110656,"data_stored":263110656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351727616,"data_stored":351727616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352890880,"data_stored":352890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:58.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:28:59.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:28:59.587 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:28:59.648 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T02:28:58.551722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":698744000,"num_objects":85303,"num_object_clones":0,"num_object_copies":341212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102254,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":34299904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102281,"ondisk_log_size":102281,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2619408,"kb_used_data":1284048,"kb_used_omap":14,"kb_used_meta":1335281,"kb_avail":374867952,"statfs":{"total":386547056640,"available":383864782848,"internally_reserved":0,"allocated":1314865152,"data_stored":1314087732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1367328307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,4,8,6,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17612800,"num_objects":-2150,"num_object_clones":0,"num_object_copies":-8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2150,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004098"},"pg_stats":[{"pgid":"1.f","version":"36'6329","reported_seq":6426,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.871132+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:57.871132+0000","last_peered":"2024-04-26T02:28:57.871132+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:57.871132+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42557440,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6418","reported_seq":6505,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.792934+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:55.792934+0000","last_peered":"2024-04-26T02:28:55.792934+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:55.792934+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44347916,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6293","reported_seq":6411,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.167721+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:54.167721+0000","last_peered":"2024-04-26T02:28:54.167721+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:54.167721+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43718942,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6292,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6247","reported_seq":6450,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.903191+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:28:55.903191+0000","last_peered":"2024-04-26T02:28:55.903191+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:55.903191+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42865164,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6324","reported_seq":6439,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.851125+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:57.851125+0000","last_peered":"2024-04-26T02:28:57.851125+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:57.851125+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43192844,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6380","reported_seq":6481,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.059197+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:58.059197+0000","last_peered":"2024-04-26T02:28:58.059197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:58.059197+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6380,"log_dups_size":0,"ondisk_log_size":6380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42762240,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6380,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6405","reported_seq":10652,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:55.625915+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:55.625915+0000","last_peered":"2024-04-26T02:28:55.625915+0000","last_clean":"2024-04-26T02:28:55.625915+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:55.625915+0000","last_undegraded":"2024-04-26T02:28:55.625915+0000","last_fullsized":"2024-04-26T02:28:55.625915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:56.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848204,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6540","reported_seq":6655,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.831886+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:55.831886+0000","last_peered":"2024-04-26T02:28:55.831886+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:55.831886+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44634636,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6407","reported_seq":6523,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.925615+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:57.925615+0000","last_peered":"2024-04-26T02:28:57.925615+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:57.925615+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43680792,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6261","reported_seq":6367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.159394+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:54.159394+0000","last_peered":"2024-04-26T02:28:54.159394+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:54.159394+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43438604,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6428","reported_seq":6528,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.035746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:58.035746+0000","last_peered":"2024-04-26T02:28:58.035746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:58.035746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43338514,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6335","reported_seq":6433,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886926+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:57.886926+0000","last_peered":"2024-04-26T02:28:57.886926+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:57.886926+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104518,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6398","reported_seq":6514,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.074514+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:54.074514+0000","last_peered":"2024-04-26T02:28:54.074514+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:54.074514+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079366,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6605","reported_seq":6697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886251+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:57.886251+0000","last_peered":"2024-04-26T02:28:57.886251+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:57.886251+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45191692,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6485","reported_seq":10690,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:57.697632+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:28:57.697632+0000","last_peered":"2024-04-26T02:28:57.697632+0000","last_clean":"2024-04-26T02:28:57.697632+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:28:57.697632+0000","last_undegraded":"2024-04-26T02:28:57.697632+0000","last_fullsized":"2024-04-26T02:28:57.697632+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:58.236100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43987468,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6426","reported_seq":6522,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.757572+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:55.757572+0000","last_peered":"2024-04-26T02:28:55.757572+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:55.757572+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43995660,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":698744000,"num_objects":85303,"num_object_clones":0,"num_object_copies":341212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102254,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":34299904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1312186368,"data_stored":1312186368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102281,"ondisk_log_size":102281,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700796,"kb_used_data":340056,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93671044,"statfs":{"total":96636764160,"available":95919149056,"internally_reserved":0,"allocated":348217344,"data_stored":348022989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696168,"kb_used_data":344284,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93675672,"statfs":{"total":96636764160,"available":95923888128,"internally_reserved":0,"allocated":352546816,"data_stored":352352461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527100,"kb_used_data":256692,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93844740,"statfs":{"total":96636764160,"available":96097013760,"internally_reserved":0,"allocated":262852608,"data_stored":262658253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695344,"kb_used_data":343016,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93676496,"statfs":{"total":96636764160,"available":95924731904,"internally_reserved":0,"allocated":351248384,"data_stored":351054029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350064640,"data_stored":350064640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262492160,"data_stored":262492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351727616,"data_stored":351727616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347901952,"data_stored":347901952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:28:59.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:00.504 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:00.505 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:00.558 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T02:28:58.551722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":698744000,"num_objects":85303,"num_object_clones":0,"num_object_copies":341212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102254,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":34299904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102281,"ondisk_log_size":102281,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2619408,"kb_used_data":1284048,"kb_used_omap":14,"kb_used_meta":1335281,"kb_avail":374867952,"statfs":{"total":386547056640,"available":383864782848,"internally_reserved":0,"allocated":1314865152,"data_stored":1314087732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1367328307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,4,8,6,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17612800,"num_objects":-2150,"num_object_clones":0,"num_object_copies":-8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2150,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004098"},"pg_stats":[{"pgid":"1.f","version":"36'6329","reported_seq":6426,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.871132+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:57.871132+0000","last_peered":"2024-04-26T02:28:57.871132+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:57.871132+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42557440,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6418","reported_seq":6505,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.792934+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:28:55.792934+0000","last_peered":"2024-04-26T02:28:55.792934+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:28:55.792934+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44347916,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6293","reported_seq":6411,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.167721+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:54.167721+0000","last_peered":"2024-04-26T02:28:54.167721+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:54.167721+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43718942,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6292,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6247","reported_seq":6450,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.903191+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:28:55.903191+0000","last_peered":"2024-04-26T02:28:55.903191+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:28:55.903191+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42865164,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6324","reported_seq":6439,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.851125+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:57.851125+0000","last_peered":"2024-04-26T02:28:57.851125+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:57.851125+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43192844,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6380","reported_seq":6481,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.059197+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:58.059197+0000","last_peered":"2024-04-26T02:28:58.059197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:58.059197+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6380,"log_dups_size":0,"ondisk_log_size":6380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42762240,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6380,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6405","reported_seq":10652,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:55.625915+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:28:55.625915+0000","last_peered":"2024-04-26T02:28:55.625915+0000","last_clean":"2024-04-26T02:28:55.625915+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:28:55.625915+0000","last_undegraded":"2024-04-26T02:28:55.625915+0000","last_fullsized":"2024-04-26T02:28:55.625915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:56.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848204,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6540","reported_seq":6655,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.831886+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:28:55.831886+0000","last_peered":"2024-04-26T02:28:55.831886+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:28:55.831886+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44634636,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6407","reported_seq":6523,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.925615+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:57.925615+0000","last_peered":"2024-04-26T02:28:57.925615+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:57.925615+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43680792,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6261","reported_seq":6367,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.159394+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:54.159394+0000","last_peered":"2024-04-26T02:28:54.159394+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:54.159394+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43438604,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6428","reported_seq":6528,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.035746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:58.035746+0000","last_peered":"2024-04-26T02:28:58.035746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:58.035746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43338514,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6335","reported_seq":6433,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886926+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:57.886926+0000","last_peered":"2024-04-26T02:28:57.886926+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:57.886926+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104518,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6398","reported_seq":6514,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:54.074514+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:54.074514+0000","last_peered":"2024-04-26T02:28:54.074514+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:54.074514+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44079366,"num_objects":5381,"num_object_clones":0,"num_object_copies":21524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6605","reported_seq":6697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886251+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:57.886251+0000","last_peered":"2024-04-26T02:28:57.886251+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:57.886251+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45191692,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6485","reported_seq":10690,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:57.697632+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:28:57.697632+0000","last_peered":"2024-04-26T02:28:57.697632+0000","last_clean":"2024-04-26T02:28:57.697632+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:28:57.697632+0000","last_undegraded":"2024-04-26T02:28:57.697632+0000","last_fullsized":"2024-04-26T02:28:57.697632+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:58.236100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43987468,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6426","reported_seq":6522,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:55.757572+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:28:55.757572+0000","last_peered":"2024-04-26T02:28:55.757572+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:28:55.757572+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43995660,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":698744000,"num_objects":85303,"num_object_clones":0,"num_object_copies":341212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102254,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":34299904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1312186368,"data_stored":1312186368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102281,"ondisk_log_size":102281,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700796,"kb_used_data":340056,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93671044,"statfs":{"total":96636764160,"available":95919149056,"internally_reserved":0,"allocated":348217344,"data_stored":348022989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696168,"kb_used_data":344284,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93675672,"statfs":{"total":96636764160,"available":95923888128,"internally_reserved":0,"allocated":352546816,"data_stored":352352461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527100,"kb_used_data":256692,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93844740,"statfs":{"total":96636764160,"available":96097013760,"internally_reserved":0,"allocated":262852608,"data_stored":262658253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695344,"kb_used_data":343016,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93676496,"statfs":{"total":96636764160,"available":95924731904,"internally_reserved":0,"allocated":351248384,"data_stored":351054029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350064640,"data_stored":350064640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262492160,"data_stored":262492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351727616,"data_stored":351727616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347901952,"data_stored":347901952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6329", 'reported_seq': 6426, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:57.871132+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:28:57.871132+0000', 'last_peered': '2024-04-26T02:28:57.871132+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:28:57.871132+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6329, 'log_dups_size': 0, 'ondisk_log_size': 6329, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42557440, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 20780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1992, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6329, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3203}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'6418", 'reported_seq': 6505, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:55.792934+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:28:55.792934+0000', 'last_peered': '2024-04-26T02:28:55.792934+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:28:55.792934+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6418, 'log_dups_size': 0, 'ondisk_log_size': 6418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44347916, 'num_objects': 5414, 'num_object_clones': 0, 'num_object_copies': 21656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2070, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6416, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6293", 'reported_seq': 6411, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:54.167721+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:28:54.167721+0000', 'last_peered': '2024-04-26T02:28:54.167721+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:28:54.167721+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6293, 'log_dups_size': 0, 'ondisk_log_size': 6293, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43718942, 'num_objects': 5338, 'num_object_clones': 0, 'num_object_copies': 21352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5338, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6292, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3331}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:00.559 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6247", 'reported_seq': 6450, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:55.903191+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:28:55.903191+0000', 'last_peered': '2024-04-26T02:28:55.903191+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:28:55.903191+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6247, 'log_dups_size': 0, 'ondisk_log_size': 6247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42865164, 'num_objects': 5233, 'num_object_clones': 0, 'num_object_copies': 20932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1901, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6245, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6324", 'reported_seq': 6439, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:57.851125+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:28:57.851125+0000', 'last_peered': '2024-04-26T02:28:57.851125+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:28:57.851125+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6324, 'log_dups_size': 0, 'ondisk_log_size': 6324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43192844, 'num_objects': 5273, 'num_object_clones': 0, 'num_object_copies': 21092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6322, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3289}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6380", 'reported_seq': 6481, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:58.059197+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:28:58.059197+0000', 'last_peered': '2024-04-26T02:28:58.059197+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:28:58.059197+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6380, 'log_dups_size': 0, 'ondisk_log_size': 6380, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42762240, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6380, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3195}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'6540", 'reported_seq': 6655, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:55.831886+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:28:55.831886+0000', 'last_peered': '2024-04-26T02:28:55.831886+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:28:55.831886+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6540, 'log_dups_size': 0, 'ondisk_log_size': 6540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44634636, 'num_objects': 5449, 'num_object_clones': 0, 'num_object_copies': 21796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6538, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6407", 'reported_seq': 6523, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:57.925615+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:28:57.925615+0000', 'last_peered': '2024-04-26T02:28:57.925615+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:28:57.925615+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6407, 'log_dups_size': 0, 'ondisk_log_size': 6407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43680792, 'num_objects': 5333, 'num_object_clones': 0, 'num_object_copies': 21332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1980, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6404, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:00.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6261", 'reported_seq': 6367, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:54.159394+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:28:54.159394+0000', 'last_peered': '2024-04-26T02:28:54.159394+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:28:54.159394+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6261, 'log_dups_size': 0, 'ondisk_log_size': 6261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43438604, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6259, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3277}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6428", 'reported_seq': 6528, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:58.035746+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:28:58.035746+0000', 'last_peered': '2024-04-26T02:28:58.035746+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:28:58.035746+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6428, 'log_dups_size': 0, 'ondisk_log_size': 6428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43338514, 'num_objects': 5291, 'num_object_clones': 0, 'num_object_copies': 21164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2001, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6425, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3293}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6335", 'reported_seq': 6433, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:57.886926+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:28:57.886926+0000', 'last_peered': '2024-04-26T02:28:57.886926+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:28:57.886926+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6335, 'log_dups_size': 0, 'ondisk_log_size': 6335, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43104518, 'num_objects': 5262, 'num_object_clones': 0, 'num_object_copies': 21048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1962, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6334, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'6398", 'reported_seq': 6514, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:54.074514+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:28:54.074514+0000', 'last_peered': '2024-04-26T02:28:54.074514+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:28:54.074514+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6398, 'log_dups_size': 0, 'ondisk_log_size': 6398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44079366, 'num_objects': 5381, 'num_object_clones': 0, 'num_object_copies': 21524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6397, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'6605", 'reported_seq': 6697, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:57.886251+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:28:57.886251+0000', 'last_peered': '2024-04-26T02:28:57.886251+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:28:57.886251+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6605, 'log_dups_size': 0, 'ondisk_log_size': 6605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45191692, 'num_objects': 5517, 'num_object_clones': 0, 'num_object_copies': 22068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6603, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:00.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'6426", 'reported_seq': 6522, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:28:55.757572+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:28:55.757572+0000', 'last_peered': '2024-04-26T02:28:55.757572+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:28:55.757572+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6426, 'log_dups_size': 0, 'ondisk_log_size': 6426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43995660, 'num_objects': 5371, 'num_object_clones': 0, 'num_object_copies': 21484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6424, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:00.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:01.537 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:01.537 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28263/340312 objects degraded (8.305%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":338,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85078,"data_bytes":696900800,"bytes_used":2685804544,"bytes_avail":383861252096,"bytes_total":386547056640,"degraded_objects":28263,"degraded_total":340312,"degraded_ratio":0.08305025976163051,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":140706,"recovering_keys_per_sec":0,"num_objects_recovered":248,"num_bytes_recovered":1236992,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:01.596 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:01.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:03.244 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:03.250 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:03.394 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-26T02:29:02.555175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693763264,"num_objects":84695,"num_object_clones":0,"num_object_copies":338780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102862,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":34897920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102889,"ondisk_log_size":102889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643696,"kb_used_data":1273648,"kb_used_omap":14,"kb_used_meta":1369969,"kb_avail":374843664,"statfs":{"total":386547056640,"available":383839911936,"internally_reserved":0,"allocated":1304215552,"data_stored":1303438132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1402848819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,8,7,8,8,3,0,0,2,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19496960,"num_objects":-2380,"num_object_clones":0,"num_object_copies":-9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006086"},"pg_stats":[{"pgid":"1.f","version":"36'6329","reported_seq":6426,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.871132+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:28:57.871132+0000","last_peered":"2024-04-26T02:28:57.871132+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:28:57.871132+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42557440,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6498","reported_seq":6586,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:00.910898+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:00.910898+0000","last_peered":"2024-04-26T02:29:00.910898+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:00.910898+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43692556,"num_objects":5334,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6365","reported_seq":6487,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:59.207968+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:28:59.207968+0000","last_peered":"2024-04-26T02:28:59.207968+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:28:59.207968+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43129118,"num_objects":5266,"num_object_clones":0,"num_object_copies":21064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6364,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6312","reported_seq":6663,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:01.000229+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:01.000229+0000","last_peered":"2024-04-26T02:29:01.000229+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:01.000229+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42332684,"num_objects":5168,"num_object_clones":0,"num_object_copies":20672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6310,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6324","reported_seq":6439,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.851125+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:28:57.851125+0000","last_peered":"2024-04-26T02:28:57.851125+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:28:57.851125+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43192844,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6380","reported_seq":6481,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.059197+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:28:58.059197+0000","last_peered":"2024-04-26T02:28:58.059197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:28:58.059197+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6380,"log_dups_size":0,"ondisk_log_size":6380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42762240,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6380,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6489","reported_seq":10736,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:00.847023+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:00.847023+0000","last_peered":"2024-04-26T02:29:00.847023+0000","last_clean":"2024-04-26T02:29:00.847023+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:00.847023+0000","last_undegraded":"2024-04-26T02:29:00.847023+0000","last_fullsized":"2024-04-26T02:29:00.847023+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:01.991327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43160076,"num_objects":5269,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6606","reported_seq":6725,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:00.948241+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:00.948241+0000","last_peered":"2024-04-26T02:29:00.948241+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:00.948241+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6606,"log_dups_size":0,"ondisk_log_size":6606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44093964,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6407","reported_seq":6523,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.925615+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:28:57.925615+0000","last_peered":"2024-04-26T02:28:57.925615+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:28:57.925615+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43680792,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6335","reported_seq":6445,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:59.192172+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:28:59.192172+0000","last_peered":"2024-04-26T02:28:59.192172+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:28:59.192172+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42832396,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6333,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6428","reported_seq":6528,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:58.035746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:28:58.035746+0000","last_peered":"2024-04-26T02:28:58.035746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:28:58.035746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43338514,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6425,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6335","reported_seq":6433,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886926+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:28:57.886926+0000","last_peered":"2024-04-26T02:28:57.886926+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:28:57.886926+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43104518,"num_objects":5262,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6477","reported_seq":6613,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:59.195847+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:28:59.195847+0000","last_peered":"2024-04-26T02:28:59.195847+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:28:59.195847+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43432198,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6476,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6605","reported_seq":6697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:28:57.886251+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:28:57.886251+0000","last_peered":"2024-04-26T02:28:57.886251+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:28:57.886251+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45191692,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6485","reported_seq":10690,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:28:57.697632+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:28:57.697632+0000","last_peered":"2024-04-26T02:28:57.697632+0000","last_clean":"2024-04-26T02:28:57.697632+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:28:57.697632+0000","last_undegraded":"2024-04-26T02:28:57.697632+0000","last_fullsized":"2024-04-26T02:28:57.697632+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:58.236100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43987468,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6514","reported_seq":6615,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:00.928827+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:00.928827+0000","last_peered":"2024-04-26T02:29:00.928827+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:00.928827+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43274764,"num_objects":5283,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6512,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":693763264,"num_objects":84695,"num_object_clones":0,"num_object_copies":338780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102862,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":34897920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1302355968,"data_stored":1302355968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102889,"ondisk_log_size":102889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700796,"kb_used_data":340056,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93671044,"statfs":{"total":96636764160,"available":95919149056,"internally_reserved":0,"allocated":348217344,"data_stored":348022989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699616,"kb_used_data":339412,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93672224,"statfs":{"total":96636764160,"available":95920357376,"internally_reserved":0,"allocated":347557888,"data_stored":347363533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,2,5,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527100,"kb_used_data":256692,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93844740,"statfs":{"total":96636764160,"available":96097013760,"internally_reserved":0,"allocated":262852608,"data_stored":262658253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716184,"kb_used_data":337488,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93655656,"statfs":{"total":96636764160,"available":95903391744,"internally_reserved":0,"allocated":345587712,"data_stored":345393357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":387772111},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":345190400,"data_stored":345190400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262492160,"data_stored":262492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346771456,"data_stored":346771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":347901952,"data_stored":347901952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:06.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:07.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:07.259 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:07.346 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-26T02:29:06.556269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":683990208,"num_objects":83502,"num_object_clones":0,"num_object_copies":334008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104055,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4677,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104082,"ondisk_log_size":104082,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686076,"kb_used_data":1259144,"kb_used_omap":14,"kb_used_meta":1426865,"kb_avail":374801284,"statfs":{"total":386547056640,"available":383796514816,"internally_reserved":0,"allocated":1289363456,"data_stored":1288586036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1461110323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,4,14,14,3,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22831104,"num_objects":-2787,"num_object_clones":0,"num_object_copies":-11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005723"},"pg_stats":[{"pgid":"1.f","version":"36'6393","reported_seq":6492,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.218361+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:03.218361+0000","last_peered":"2024-04-26T02:29:03.218361+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:03.218361+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42033152,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6577","reported_seq":6673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.155316+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:06.155316+0000","last_peered":"2024-04-26T02:29:06.155316+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:06.155316+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6577,"log_dups_size":0,"ondisk_log_size":6577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43045388,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6575,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6434","reported_seq":6555,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.184234+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:04.184234+0000","last_peered":"2024-04-26T02:29:04.184234+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:04.184234+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563870,"num_objects":5197,"num_object_clones":0,"num_object_copies":20788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6433,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6392","reported_seq":6879,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.129155+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:06.129155+0000","last_peered":"2024-04-26T02:29:06.129155+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:06.129155+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677324,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6393","reported_seq":6512,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.076130+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:03.076130+0000","last_peered":"2024-04-26T02:29:03.076130+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:03.076130+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42627596,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6460","reported_seq":6571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.160062+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:03.160062+0000","last_peered":"2024-04-26T02:29:03.160062+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:03.160062+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42106880,"num_objects":5140,"num_object_clones":0,"num_object_copies":20560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6569","reported_seq":10816,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:06.158161+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:06.158161+0000","last_peered":"2024-04-26T02:29:06.158161+0000","last_clean":"2024-04-26T02:29:06.158161+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:06.158161+0000","last_undegraded":"2024-04-26T02:29:06.158161+0000","last_fullsized":"2024-04-26T02:29:06.158161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:07.043323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42504716,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6701","reported_seq":6821,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.114988+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:06.114988+0000","last_peered":"2024-04-26T02:29:06.114988+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:06.114988+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43315724,"num_objects":5288,"num_object_clones":0,"num_object_copies":21152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6485","reported_seq":6607,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.021971+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:03.021971+0000","last_peered":"2024-04-26T02:29:03.021971+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:03.021971+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43041816,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6402","reported_seq":6514,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.070783+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:04.070783+0000","last_peered":"2024-04-26T02:29:04.070783+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:04.070783+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42283532,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6497","reported_seq":6604,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.202395+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:03.202395+0000","last_peered":"2024-04-26T02:29:03.202395+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:03.202395+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6497,"log_dups_size":0,"ondisk_log_size":6497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42773266,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6494,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6414","reported_seq":6515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.102961+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:03.102961+0000","last_peered":"2024-04-26T02:29:03.102961+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:03.102961+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42457350,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3224}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6548","reported_seq":6686,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.159113+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:04.159113+0000","last_peered":"2024-04-26T02:29:04.159113+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:04.159113+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6548,"log_dups_size":0,"ondisk_log_size":6548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42850566,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6547,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6679","reported_seq":6775,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.164596+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:03.164596+0000","last_peered":"2024-04-26T02:29:03.164596+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:03.164596+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44585484,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3391}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6558","reported_seq":10763,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:02.584514+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:02.584514+0000","last_peered":"2024-04-26T02:29:02.584514+0000","last_clean":"2024-04-26T02:29:02.584514+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:02.584514+0000","last_undegraded":"2024-04-26T02:29:02.584514+0000","last_fullsized":"2024-04-26T02:29:02.584514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:03.315382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43389452,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6580","reported_seq":6682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.009162+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:06.009162+0000","last_peered":"2024-04-26T02:29:06.009162+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:06.009162+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734092,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":683990208,"num_objects":83502,"num_object_clones":0,"num_object_copies":334008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104055,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4677,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1287102464,"data_stored":1287102464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104082,"ondisk_log_size":104082,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722668,"kb_used_data":335576,"kb_used_omap":4,"kb_used_meta":387067,"kb_avail":93649172,"statfs":{"total":96636764160,"available":95896752128,"internally_reserved":0,"allocated":343629824,"data_stored":343435469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":396357328},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,6,2,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717252,"kb_used_data":334844,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93654588,"statfs":{"total":96636764160,"available":95902298112,"internally_reserved":0,"allocated":342880256,"data_stored":342685901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530748,"kb_used_data":256180,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93841092,"statfs":{"total":96636764160,"available":96093278208,"internally_reserved":0,"allocated":262328320,"data_stored":262133965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715408,"kb_used_data":332544,"kb_used_omap":4,"kb_used_meta":382843,"kb_avail":93656432,"statfs":{"total":96636764160,"available":95904186368,"internally_reserved":0,"allocated":340525056,"data_stored":340330701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":392031951},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339992576,"data_stored":339992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261947392,"data_stored":261947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342032384,"data_stored":342032384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343130112,"data_stored":343130112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:07.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:08.523 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:08.525 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:08.603 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-26T02:29:06.556269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":683990208,"num_objects":83502,"num_object_clones":0,"num_object_copies":334008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104055,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4677,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104082,"ondisk_log_size":104082,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686076,"kb_used_data":1259144,"kb_used_omap":14,"kb_used_meta":1426865,"kb_avail":374801284,"statfs":{"total":386547056640,"available":383796514816,"internally_reserved":0,"allocated":1289363456,"data_stored":1288586036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1461110323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,4,14,14,3,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22831104,"num_objects":-2787,"num_object_clones":0,"num_object_copies":-11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005723"},"pg_stats":[{"pgid":"1.f","version":"36'6393","reported_seq":6492,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.218361+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:03.218361+0000","last_peered":"2024-04-26T02:29:03.218361+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:03.218361+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42033152,"num_objects":5131,"num_object_clones":0,"num_object_copies":20524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6577","reported_seq":6673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.155316+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:06.155316+0000","last_peered":"2024-04-26T02:29:06.155316+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:06.155316+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6577,"log_dups_size":0,"ondisk_log_size":6577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43045388,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6575,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6434","reported_seq":6555,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.184234+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:04.184234+0000","last_peered":"2024-04-26T02:29:04.184234+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:04.184234+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563870,"num_objects":5197,"num_object_clones":0,"num_object_copies":20788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6433,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6392","reported_seq":6879,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.129155+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:06.129155+0000","last_peered":"2024-04-26T02:29:06.129155+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:06.129155+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677324,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6393","reported_seq":6512,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.076130+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:03.076130+0000","last_peered":"2024-04-26T02:29:03.076130+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:03.076130+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42627596,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6460","reported_seq":6571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.160062+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:03.160062+0000","last_peered":"2024-04-26T02:29:03.160062+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:03.160062+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42106880,"num_objects":5140,"num_object_clones":0,"num_object_copies":20560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6569","reported_seq":10816,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:06.158161+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:06.158161+0000","last_peered":"2024-04-26T02:29:06.158161+0000","last_clean":"2024-04-26T02:29:06.158161+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:06.158161+0000","last_undegraded":"2024-04-26T02:29:06.158161+0000","last_fullsized":"2024-04-26T02:29:06.158161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:07.043323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42504716,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6701","reported_seq":6821,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.114988+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:06.114988+0000","last_peered":"2024-04-26T02:29:06.114988+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:06.114988+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43315724,"num_objects":5288,"num_object_clones":0,"num_object_copies":21152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6485","reported_seq":6607,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.021971+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:03.021971+0000","last_peered":"2024-04-26T02:29:03.021971+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:03.021971+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43041816,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6402","reported_seq":6514,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.070783+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:04.070783+0000","last_peered":"2024-04-26T02:29:04.070783+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:04.070783+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42283532,"num_objects":5162,"num_object_clones":0,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6497","reported_seq":6604,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.202395+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:03.202395+0000","last_peered":"2024-04-26T02:29:03.202395+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:03.202395+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6497,"log_dups_size":0,"ondisk_log_size":6497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42773266,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6494,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6414","reported_seq":6515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.102961+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:03.102961+0000","last_peered":"2024-04-26T02:29:03.102961+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:03.102961+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42457350,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3224}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6548","reported_seq":6686,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:04.159113+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:04.159113+0000","last_peered":"2024-04-26T02:29:04.159113+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:04.159113+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6548,"log_dups_size":0,"ondisk_log_size":6548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42850566,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6547,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6679","reported_seq":6775,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:03.164596+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:03.164596+0000","last_peered":"2024-04-26T02:29:03.164596+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:03.164596+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44585484,"num_objects":5443,"num_object_clones":0,"num_object_copies":21772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6677,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3391}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6558","reported_seq":10763,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:02.584514+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:02.584514+0000","last_peered":"2024-04-26T02:29:02.584514+0000","last_clean":"2024-04-26T02:29:02.584514+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:02.584514+0000","last_undegraded":"2024-04-26T02:29:02.584514+0000","last_fullsized":"2024-04-26T02:29:02.584514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:03.315382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43389452,"num_objects":5297,"num_object_clones":0,"num_object_copies":21188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6580","reported_seq":6682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.009162+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:06.009162+0000","last_peered":"2024-04-26T02:29:06.009162+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:06.009162+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734092,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":683990208,"num_objects":83502,"num_object_clones":0,"num_object_copies":334008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104055,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4677,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1287102464,"data_stored":1287102464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104082,"ondisk_log_size":104082,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722668,"kb_used_data":335576,"kb_used_omap":4,"kb_used_meta":387067,"kb_avail":93649172,"statfs":{"total":96636764160,"available":95896752128,"internally_reserved":0,"allocated":343629824,"data_stored":343435469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":396357328},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,6,2,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717252,"kb_used_data":334844,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93654588,"statfs":{"total":96636764160,"available":95902298112,"internally_reserved":0,"allocated":342880256,"data_stored":342685901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530748,"kb_used_data":256180,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93841092,"statfs":{"total":96636764160,"available":96093278208,"internally_reserved":0,"allocated":262328320,"data_stored":262133965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715408,"kb_used_data":332544,"kb_used_omap":4,"kb_used_meta":382843,"kb_avail":93656432,"statfs":{"total":96636764160,"available":95904186368,"internally_reserved":0,"allocated":340525056,"data_stored":340330701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":392031951},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339992576,"data_stored":339992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261947392,"data_stored":261947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342032384,"data_stored":342032384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343130112,"data_stored":343130112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6393", 'reported_seq': 6492, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.218361+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:03.218361+0000', 'last_peered': '2024-04-26T02:29:03.218361+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:03.218361+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6393, 'log_dups_size': 0, 'ondisk_log_size': 6393, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42033152, 'num_objects': 5131, 'num_object_clones': 0, 'num_object_copies': 20524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6393, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3140}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'6577", 'reported_seq': 6673, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:06.155316+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:06.155316+0000', 'last_peered': '2024-04-26T02:29:06.155316+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:06.155316+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6577, 'log_dups_size': 0, 'ondisk_log_size': 6577, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43045388, 'num_objects': 5255, 'num_object_clones': 0, 'num_object_copies': 21020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6575, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6434", 'reported_seq': 6555, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:04.184234+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:04.184234+0000', 'last_peered': '2024-04-26T02:29:04.184234+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:04.184234+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6434, 'log_dups_size': 0, 'ondisk_log_size': 6434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42563870, 'num_objects': 5197, 'num_object_clones': 0, 'num_object_copies': 20788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5197, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6433, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:08.605 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6392", 'reported_seq': 6879, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:06.129155+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:06.129155+0000', 'last_peered': '2024-04-26T02:29:06.129155+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:06.129155+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6392, 'log_dups_size': 0, 'ondisk_log_size': 6392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41677324, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1759, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6390, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 211, 'num_bytes_recovered': 1581056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6393", 'reported_seq': 6512, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.076130+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:03.076130+0000', 'last_peered': '2024-04-26T02:29:03.076130+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:03.076130+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6393, 'log_dups_size': 0, 'ondisk_log_size': 6393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42627596, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1984, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6391, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3222}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6460", 'reported_seq': 6571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.160062+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:03.160062+0000', 'last_peered': '2024-04-26T02:29:03.160062+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:03.160062+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6460, 'log_dups_size': 0, 'ondisk_log_size': 6460, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42106880, 'num_objects': 5140, 'num_object_clones': 0, 'num_object_copies': 20560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6460, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'6701", 'reported_seq': 6821, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:06.114988+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:06.114988+0000', 'last_peered': '2024-04-26T02:29:06.114988+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:06.114988+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6701, 'log_dups_size': 0, 'ondisk_log_size': 6701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43315724, 'num_objects': 5288, 'num_object_clones': 0, 'num_object_copies': 21152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6699, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3172}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6485", 'reported_seq': 6607, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.021971+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:03.021971+0000', 'last_peered': '2024-04-26T02:29:03.021971+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:03.021971+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6485, 'log_dups_size': 0, 'ondisk_log_size': 6485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43041816, 'num_objects': 5255, 'num_object_clones': 0, 'num_object_copies': 21020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6482, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:08.606 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6402", 'reported_seq': 6514, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:04.070783+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:04.070783+0000', 'last_peered': '2024-04-26T02:29:04.070783+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:04.070783+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6402, 'log_dups_size': 0, 'ondisk_log_size': 6402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42283532, 'num_objects': 5162, 'num_object_clones': 0, 'num_object_copies': 20648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6400, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3139}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6497", 'reported_seq': 6604, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.202395+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:03.202395+0000', 'last_peered': '2024-04-26T02:29:03.202395+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:03.202395+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6497, 'log_dups_size': 0, 'ondisk_log_size': 6497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42773266, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1998, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6494, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3227}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6414", 'reported_seq': 6515, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.102961+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:03.102961+0000', 'last_peered': '2024-04-26T02:29:03.102961+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:03.102961+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6414, 'log_dups_size': 0, 'ondisk_log_size': 6414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42457350, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 20732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6413, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3224}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'6548", 'reported_seq': 6686, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:04.159113+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:04.159113+0000', 'last_peered': '2024-04-26T02:29:04.159113+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:04.159113+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6548, 'log_dups_size': 0, 'ondisk_log_size': 6548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42850566, 'num_objects': 5231, 'num_object_clones': 0, 'num_object_copies': 20924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6547, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'6679", 'reported_seq': 6775, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:03.164596+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:03.164596+0000', 'last_peered': '2024-04-26T02:29:03.164596+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:03.164596+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6679, 'log_dups_size': 0, 'ondisk_log_size': 6679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44585484, 'num_objects': 5443, 'num_object_clones': 0, 'num_object_copies': 21772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6677, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:08.607 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'6580", 'reported_seq': 6682, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:06.009162+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:06.009162+0000', 'last_peered': '2024-04-26T02:29:06.009162+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:06.009162+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6580, 'log_dups_size': 0, 'ondisk_log_size': 6580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42734092, 'num_objects': 5217, 'num_object_clones': 0, 'num_object_copies': 20868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6578, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:08.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:10.027 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:10.027 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28076/331532 objects degraded (8.469%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82883,"data_bytes":678919360,"bytes_used":2719117312,"bytes_avail":383827939328,"bytes_total":386547056640,"degraded_objects":28076,"degraded_total":331532,"degraded_ratio":0.084685641205072215,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":96211,"recovering_keys_per_sec":0,"num_objects_recovered":242,"num_bytes_recovered":1155072,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:10.044 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:10.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:10.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:10.858 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:10.936 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-26T02:29:10.557324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676928704,"num_objects":82640,"num_object_clones":0,"num_object_copies":330560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104917,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4692,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104944,"ondisk_log_size":104944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2638092,"kb_used_data":1248596,"kb_used_omap":14,"kb_used_meta":1389425,"kb_avail":374849268,"statfs":{"total":386547056640,"available":383845650432,"internally_reserved":0,"allocated":1278562304,"data_stored":1277784884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1422771763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,6,12,8,4,3,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21815296,"num_objects":-2663,"num_object_clones":0,"num_object_copies":-10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005601"},"pg_stats":[{"pgid":"1.f","version":"36'6476","reported_seq":6575,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.349800+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:08.349800+0000","last_peered":"2024-04-26T02:29:08.349800+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:08.349800+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6476,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6577","reported_seq":6673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.155316+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:06.155316+0000","last_peered":"2024-04-26T02:29:06.155316+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:06.155316+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6577,"log_dups_size":0,"ondisk_log_size":6577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43045388,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6575,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6508","reported_seq":6632,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:09.289149+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:09.289149+0000","last_peered":"2024-04-26T02:29:09.289149+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:09.289149+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6508,"log_dups_size":0,"ondisk_log_size":6508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41957662,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6507,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6392","reported_seq":6879,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.129155+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:06.129155+0000","last_peered":"2024-04-26T02:29:06.129155+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:06.129155+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677324,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6477","reported_seq":6598,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.176184+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:08.176184+0000","last_peered":"2024-04-26T02:29:08.176184+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:08.176184+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41939468,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6475,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6535","reported_seq":6657,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.251366+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:08.251366+0000","last_peered":"2024-04-26T02:29:08.251366+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:08.251366+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6535,"log_dups_size":0,"ondisk_log_size":6535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41492480,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6569","reported_seq":10816,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:06.158161+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:06.158161+0000","last_peered":"2024-04-26T02:29:06.158161+0000","last_clean":"2024-04-26T02:29:06.158161+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:06.158161+0000","last_undegraded":"2024-04-26T02:29:06.158161+0000","last_fullsized":"2024-04-26T02:29:06.158161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:07.043323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42504716,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6701","reported_seq":6821,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.114988+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:06.114988+0000","last_peered":"2024-04-26T02:29:06.114988+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:06.114988+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43315724,"num_objects":5288,"num_object_clones":0,"num_object_copies":21152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6554","reported_seq":6682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:07.924574+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:07.924574+0000","last_peered":"2024-04-26T02:29:07.924574+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:07.924574+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42476568,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6486","reported_seq":6598,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:09.286933+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:09.286933+0000","last_peered":"2024-04-26T02:29:09.286933+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:09.286933+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41595404,"num_objects":5078,"num_object_clones":0,"num_object_copies":20312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6574","reported_seq":6681,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.137528+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:08.137528+0000","last_peered":"2024-04-26T02:29:08.137528+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:08.137528+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6574,"log_dups_size":0,"ondisk_log_size":6574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42142482,"num_objects":5145,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6486","reported_seq":6590,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.137833+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:08.137833+0000","last_peered":"2024-04-26T02:29:08.137833+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:08.137833+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41867526,"num_objects":5111,"num_object_clones":0,"num_object_copies":20444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6485,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6633","reported_seq":6771,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:09.269525+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:09.269525+0000","last_peered":"2024-04-26T02:29:09.269525+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:09.269525+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42154246,"num_objects":5146,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6632,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6756","reported_seq":6858,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:08.335376+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:08.335376+0000","last_peered":"2024-04-26T02:29:08.335376+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:08.335376+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43954700,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3317}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6640","reported_seq":10845,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:07.894047+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:07.894047+0000","last_peered":"2024-04-26T02:29:07.894047+0000","last_clean":"2024-04-26T02:29:07.894047+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:07.894047+0000","last_undegraded":"2024-04-26T02:29:07.894047+0000","last_fullsized":"2024-04-26T02:29:07.894047+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:09.292466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42717708,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6580","reported_seq":6682,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:06.009162+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:06.009162+0000","last_peered":"2024-04-26T02:29:06.009162+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:06.009162+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42734092,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":676928704,"num_objects":82640,"num_object_clones":0,"num_object_copies":330560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104917,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4692,"num_bytes_recovered":35454976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1276088320,"data_stored":1276088320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104944,"ondisk_log_size":104944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705220,"kb_used_data":330612,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93666620,"statfs":{"total":96636764160,"available":95914618880,"internally_reserved":0,"allocated":338546688,"data_stored":338352333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":383577808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,7,3,2,3,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699956,"kb_used_data":330020,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93671884,"statfs":{"total":96636764160,"available":95920009216,"internally_reserved":0,"allocated":337940480,"data_stored":337746125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,3,3,0,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517508,"kb_used_data":255420,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93854332,"statfs":{"total":96636764160,"available":96106835968,"internally_reserved":0,"allocated":261550080,"data_stored":261355725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715408,"kb_used_data":332544,"kb_used_omap":4,"kb_used_meta":382843,"kb_avail":93656432,"statfs":{"total":96636764160,"available":95904186368,"internally_reserved":0,"allocated":340525056,"data_stored":340330701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":392031951},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339992576,"data_stored":339992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261173248,"data_stored":261173248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336969728,"data_stored":336969728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":337952768,"data_stored":337952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:13.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:15.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:15.142 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:15.214 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-04-26T02:29:14.558344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667434176,"num_objects":81481,"num_object_clones":0,"num_object_copies":325924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106076,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4788,"num_bytes_recovered":35905536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106103,"ondisk_log_size":106103,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686068,"kb_used_data":1234128,"kb_used_omap":14,"kb_used_meta":1451889,"kb_avail":374801292,"statfs":{"total":386547056640,"available":383796523008,"internally_reserved":0,"allocated":1263747072,"data_stored":1262969652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1486734899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,4,6,8,4,3,0,0,1,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26329088,"num_objects":-3214,"num_object_clones":0,"num_object_copies":-12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"1.f","version":"36'6536","reported_seq":6634,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.334280+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:13.334280+0000","last_peered":"2024-04-26T02:29:13.334280+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:13.334280+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6652","reported_seq":6763,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:11.235253+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:11.235253+0000","last_peered":"2024-04-26T02:29:11.235253+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:11.235253+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42430988,"num_objects":5180,"num_object_clones":0,"num_object_copies":20720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6586","reported_seq":6712,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.249566+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:14.249566+0000","last_peered":"2024-04-26T02:29:14.249566+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:14.249566+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41318686,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6585,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6457","reported_seq":7054,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:11.230294+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:11.230294+0000","last_peered":"2024-04-26T02:29:11.230294+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:11.230294+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41144844,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6550","reported_seq":6675,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.335610+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:13.335610+0000","last_peered":"2024-04-26T02:29:13.335610+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:13.335610+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41341452,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6615","reported_seq":6748,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.210099+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:13.210099+0000","last_peered":"2024-04-26T02:29:13.210099+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:13.210099+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40837120,"num_objects":4985,"num_object_clones":0,"num_object_copies":19940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6650","reported_seq":10897,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:11.162291+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:11.162291+0000","last_peered":"2024-04-26T02:29:11.162291+0000","last_clean":"2024-04-26T02:29:11.162291+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:11.162291+0000","last_undegraded":"2024-04-26T02:29:11.162291+0000","last_fullsized":"2024-04-26T02:29:11.162291+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6650,"log_dups_size":0,"ondisk_log_size":6650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:12.981532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841164,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6648,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6773","reported_seq":6910,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:11.211446+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:11.211446+0000","last_peered":"2024-04-26T02:29:11.211446+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:11.211446+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6773,"log_dups_size":0,"ondisk_log_size":6773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42725900,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6627","reported_seq":6763,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.388855+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:13.388855+0000","last_peered":"2024-04-26T02:29:13.388855+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:13.388855+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6627,"log_dups_size":0,"ondisk_log_size":6627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41878552,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6546","reported_seq":6660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.246989+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:14.246989+0000","last_peered":"2024-04-26T02:29:14.246989+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:14.246989+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41103884,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2996}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6642","reported_seq":6751,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.314182+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:13.314182+0000","last_peered":"2024-04-26T02:29:13.314182+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:13.314182+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41585426,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6566","reported_seq":6676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.386854+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:13.386854+0000","last_peered":"2024-04-26T02:29:13.386854+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:13.386854+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41212166,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6706","reported_seq":6846,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.346214+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:14.346214+0000","last_peered":"2024-04-26T02:29:14.346214+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:14.346214+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41556230,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6833","reported_seq":6939,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.319898+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:13.319898+0000","last_peered":"2024-04-26T02:29:13.319898+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:13.319898+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43323916,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6719","reported_seq":10924,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:12.848595+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:12.848595+0000","last_peered":"2024-04-26T02:29:12.848595+0000","last_clean":"2024-04-26T02:29:12.848595+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:12.848595+0000","last_undegraded":"2024-04-26T02:29:12.848595+0000","last_fullsized":"2024-04-26T02:29:12.848595+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:14.304680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42070540,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6645","reported_seq":6759,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:11.185329+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:11.185329+0000","last_peered":"2024-04-26T02:29:11.185329+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:11.185329+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42201612,"num_objects":5152,"num_object_clones":0,"num_object_copies":20608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3110}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":667434176,"num_objects":81481,"num_object_clones":0,"num_object_copies":325924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106076,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4788,"num_bytes_recovered":35905536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1261166592,"data_stored":1261166592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106103,"ondisk_log_size":106103,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":325940,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":333762560,"data_stored":333568205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717488,"kb_used_data":325352,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93654352,"statfs":{"total":96636764160,"available":95902056448,"internally_reserved":0,"allocated":333160448,"data_stored":332966093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525252,"kb_used_data":254840,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93846588,"statfs":{"total":96636764160,"available":96098906112,"internally_reserved":0,"allocated":260956160,"data_stored":260761805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720580,"kb_used_data":327996,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93651260,"statfs":{"total":96636764160,"available":95898890240,"internally_reserved":0,"allocated":335867904,"data_stored":335673549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,2,3,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335306752,"data_stored":335306752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260558848,"data_stored":260558848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332173312,"data_stored":332173312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333127680,"data_stored":333127680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:15.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:16.628 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:16.629 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:16.703 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-04-26T02:29:16.559330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664386752,"num_objects":81109,"num_object_clones":0,"num_object_copies":324436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81109,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106448,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4871,"num_bytes_recovered":36421632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106475,"ondisk_log_size":106475,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685520,"kb_used_data":1229412,"kb_used_omap":14,"kb_used_meta":1456049,"kb_avail":374801840,"statfs":{"total":386547056640,"available":383797084160,"internally_reserved":0,"allocated":1258917888,"data_stored":1258140468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1490994739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,9,8,12,1,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22880256,"num_objects":-2793,"num_object_clones":0,"num_object_copies":-11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003621"},"pg_stats":[{"pgid":"1.f","version":"36'6536","reported_seq":6634,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.334280+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:13.334280+0000","last_peered":"2024-04-26T02:29:13.334280+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:13.334280+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6733","reported_seq":6854,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.189286+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:16.189286+0000","last_peered":"2024-04-26T02:29:16.189286+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:16.189286+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41767436,"num_objects":5099,"num_object_clones":0,"num_object_copies":20396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6731,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6586","reported_seq":6712,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.249566+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:14.249566+0000","last_peered":"2024-04-26T02:29:14.249566+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:14.249566+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41318686,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6585,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6532","reported_seq":7257,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.257169+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:16.257169+0000","last_peered":"2024-04-26T02:29:16.257169+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:16.257169+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40530444,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":2547712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6550","reported_seq":6675,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.335610+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:13.335610+0000","last_peered":"2024-04-26T02:29:13.335610+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:13.335610+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41341452,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6615","reported_seq":6748,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.210099+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:13.210099+0000","last_peered":"2024-04-26T02:29:13.210099+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:13.210099+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40837120,"num_objects":4985,"num_object_clones":0,"num_object_copies":19940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6718","reported_seq":10965,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:16.164903+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:16.164903+0000","last_peered":"2024-04-26T02:29:16.164903+0000","last_clean":"2024-04-26T02:29:16.164903+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:16.164903+0000","last_undegraded":"2024-04-26T02:29:16.164903+0000","last_fullsized":"2024-04-26T02:29:16.164903+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:18.982691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284108,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6857","reported_seq":7000,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.074330+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:16.074330+0000","last_peered":"2024-04-26T02:29:16.074330+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:16.074330+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42037772,"num_objects":5132,"num_object_clones":0,"num_object_copies":20528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6627","reported_seq":6763,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.388855+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:13.388855+0000","last_peered":"2024-04-26T02:29:13.388855+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:13.388855+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6627,"log_dups_size":0,"ondisk_log_size":6627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41878552,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6546","reported_seq":6660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.246989+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:14.246989+0000","last_peered":"2024-04-26T02:29:14.246989+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:14.246989+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41103884,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2996}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6642","reported_seq":6751,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.314182+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:13.314182+0000","last_peered":"2024-04-26T02:29:13.314182+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:13.314182+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41585426,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6566","reported_seq":6676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.386854+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:13.386854+0000","last_peered":"2024-04-26T02:29:13.386854+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:13.386854+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41212166,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6706","reported_seq":6846,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.346214+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:14.346214+0000","last_peered":"2024-04-26T02:29:14.346214+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:14.346214+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41556230,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6833","reported_seq":6939,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.319898+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:13.319898+0000","last_peered":"2024-04-26T02:29:13.319898+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:13.319898+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43323916,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6719","reported_seq":10924,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:12.848595+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:12.848595+0000","last_peered":"2024-04-26T02:29:12.848595+0000","last_clean":"2024-04-26T02:29:12.848595+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:12.848595+0000","last_undegraded":"2024-04-26T02:29:12.848595+0000","last_fullsized":"2024-04-26T02:29:12.848595+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:14.304680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42070540,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6709","reported_seq":6843,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.146862+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:16.146862+0000","last_peered":"2024-04-26T02:29:16.146862+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:16.146862+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677324,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":664386752,"num_objects":81109,"num_object_clones":0,"num_object_copies":324436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81109,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106448,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4871,"num_bytes_recovered":36421632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1256312832,"data_stored":1256312832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106475,"ondisk_log_size":106475,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":325940,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":333762560,"data_stored":333568205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717488,"kb_used_data":325352,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93654352,"statfs":{"total":96636764160,"available":95902056448,"internally_reserved":0,"allocated":333160448,"data_stored":332966093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525252,"kb_used_data":254840,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93846588,"statfs":{"total":96636764160,"available":96098906112,"internally_reserved":0,"allocated":260956160,"data_stored":260761805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720032,"kb_used_data":323280,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93651808,"statfs":{"total":96636764160,"available":95899451392,"internally_reserved":0,"allocated":331038720,"data_stored":330844365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,5,6,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330452992,"data_stored":330452992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260558848,"data_stored":260558848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332173312,"data_stored":332173312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333127680,"data_stored":333127680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:16.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6536", 'reported_seq': 6634, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.334280+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:13.334280+0000', 'last_peered': '2024-04-26T02:29:13.334280+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:13.334280+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6536, 'log_dups_size': 0, 'ondisk_log_size': 6536, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40861696, 'num_objects': 4988, 'num_object_clones': 0, 'num_object_copies': 19952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6536, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'6733", 'reported_seq': 6854, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:16.189286+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:16.189286+0000', 'last_peered': '2024-04-26T02:29:16.189286+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:16.189286+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6733, 'log_dups_size': 0, 'ondisk_log_size': 6733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41767436, 'num_objects': 5099, 'num_object_clones': 0, 'num_object_copies': 20396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6731, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3048}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6586", 'reported_seq': 6712, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:14.249566+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:14.249566+0000', 'last_peered': '2024-04-26T02:29:14.249566+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:14.249566+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6586, 'log_dups_size': 0, 'ondisk_log_size': 6586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41318686, 'num_objects': 5045, 'num_object_clones': 0, 'num_object_copies': 20180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5045, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6585, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3042}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6532", 'reported_seq': 7257, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:16.257169+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:16.257169+0000', 'last_peered': '2024-04-26T02:29:16.257169+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:16.257169+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6532, 'log_dups_size': 0, 'ondisk_log_size': 6532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40530444, 'num_objects': 4948, 'num_object_clones': 0, 'num_object_copies': 19792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1639, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6530, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 331, 'num_bytes_recovered': 2547712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:16.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6550", 'reported_seq': 6675, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.335610+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:13.335610+0000', 'last_peered': '2024-04-26T02:29:13.335610+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:13.335610+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6550, 'log_dups_size': 0, 'ondisk_log_size': 6550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41341452, 'num_objects': 5047, 'num_object_clones': 0, 'num_object_copies': 20188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6548, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3068}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6615", 'reported_seq': 6748, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.210099+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:13.210099+0000', 'last_peered': '2024-04-26T02:29:13.210099+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:13.210099+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6615, 'log_dups_size': 0, 'ondisk_log_size': 6615, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40837120, 'num_objects': 4985, 'num_object_clones': 0, 'num_object_copies': 19940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2009, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6615, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2976}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'6857", 'reported_seq': 7000, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:16.074330+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:16.074330+0000', 'last_peered': '2024-04-26T02:29:16.074330+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:16.074330+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6857, 'log_dups_size': 0, 'ondisk_log_size': 6857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42037772, 'num_objects': 5132, 'num_object_clones': 0, 'num_object_copies': 20528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2107, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6855, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6627", 'reported_seq': 6763, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.388855+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:13.388855+0000', 'last_peered': '2024-04-26T02:29:13.388855+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:13.388855+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6627, 'log_dups_size': 0, 'ondisk_log_size': 6627, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41878552, 'num_objects': 5113, 'num_object_clones': 0, 'num_object_copies': 20452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6624, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6546", 'reported_seq': 6660, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:14.246989+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:14.246989+0000', 'last_peered': '2024-04-26T02:29:14.246989+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:14.246989+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6546, 'log_dups_size': 0, 'ondisk_log_size': 6546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41103884, 'num_objects': 5018, 'num_object_clones': 0, 'num_object_copies': 20072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6544, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2996}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:16.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6642", 'reported_seq': 6751, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.314182+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:13.314182+0000', 'last_peered': '2024-04-26T02:29:13.314182+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:13.314182+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6642, 'log_dups_size': 0, 'ondisk_log_size': 6642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41585426, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6639, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3083}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6566", 'reported_seq': 6676, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.386854+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:13.386854+0000', 'last_peered': '2024-04-26T02:29:13.386854+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:13.386854+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6566, 'log_dups_size': 0, 'ondisk_log_size': 6566, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41212166, 'num_objects': 5031, 'num_object_clones': 0, 'num_object_copies': 20124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6565, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'6706", 'reported_seq': 6846, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:14.346214+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:14.346214+0000', 'last_peered': '2024-04-26T02:29:14.346214+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:14.346214+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6706, 'log_dups_size': 0, 'ondisk_log_size': 6706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41556230, 'num_objects': 5073, 'num_object_clones': 0, 'num_object_copies': 20292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2082, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6705, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'6833", 'reported_seq': 6939, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:13.319898+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:13.319898+0000', 'last_peered': '2024-04-26T02:29:13.319898+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:13.319898+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6833, 'log_dups_size': 0, 'ondisk_log_size': 6833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43323916, 'num_objects': 5289, 'num_object_clones': 0, 'num_object_copies': 21156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6831, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:16.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'6709", 'reported_seq': 6843, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:16.146862+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:16.146862+0000', 'last_peered': '2024-04-26T02:29:16.146862+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:16.146862+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6709, 'log_dups_size': 0, 'ondisk_log_size': 6709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41677324, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6707, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3056}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:16.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:18.280 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:18.280 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27896/324436 objects degraded (8.598%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":354,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81109,"data_bytes":664386752,"bytes_used":2749972480,"bytes_avail":383797084160,"bytes_total":386547056640,"degraded_objects":27896,"degraded_total":324436,"degraded_ratio":0.085983059833064146,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":126937,"recovering_keys_per_sec":0,"num_objects_recovered":267,"num_bytes_recovered":1523712,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":232},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:18.331 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:18.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:19.642 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:19.642 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:19.659 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-26T02:29:18.559837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663796928,"num_objects":81037,"num_object_clones":0,"num_object_copies":324148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106520,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4871,"num_bytes_recovered":36421632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106547,"ondisk_log_size":106547,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707200,"kb_used_data":1228888,"kb_used_omap":14,"kb_used_meta":1478257,"kb_avail":374780160,"statfs":{"total":386547056640,"available":383774883840,"internally_reserved":0,"allocated":1258381312,"data_stored":1257603892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1513735731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,10,7,13,3,0,0,0,0,5,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20193280,"num_objects":-2465,"num_object_clones":0,"num_object_copies":-9860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003568"},"pg_stats":[{"pgid":"1.f","version":"36'6536","reported_seq":6634,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.334280+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:13.334280+0000","last_peered":"2024-04-26T02:29:13.334280+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:13.334280+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6733","reported_seq":6854,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.189286+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:16.189286+0000","last_peered":"2024-04-26T02:29:16.189286+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:16.189286+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41767436,"num_objects":5099,"num_object_clones":0,"num_object_copies":20396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6731,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6586","reported_seq":6712,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.249566+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:14.249566+0000","last_peered":"2024-04-26T02:29:14.249566+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:14.249566+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41318686,"num_objects":5045,"num_object_clones":0,"num_object_copies":20180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6585,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6532","reported_seq":7257,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.257169+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:16.257169+0000","last_peered":"2024-04-26T02:29:16.257169+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:16.257169+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40530444,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":2547712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6550","reported_seq":6675,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.335610+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:13.335610+0000","last_peered":"2024-04-26T02:29:13.335610+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:13.335610+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41341452,"num_objects":5047,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6615","reported_seq":6748,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.210099+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:13.210099+0000","last_peered":"2024-04-26T02:29:13.210099+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:13.210099+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40837120,"num_objects":4985,"num_object_clones":0,"num_object_copies":19940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6718","reported_seq":10965,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:16.164903+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:16.164903+0000","last_peered":"2024-04-26T02:29:16.164903+0000","last_clean":"2024-04-26T02:29:16.164903+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:16.164903+0000","last_undegraded":"2024-04-26T02:29:16.164903+0000","last_fullsized":"2024-04-26T02:29:16.164903+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:18.982691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284108,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6857","reported_seq":7000,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.074330+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:16.074330+0000","last_peered":"2024-04-26T02:29:16.074330+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:16.074330+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42037772,"num_objects":5132,"num_object_clones":0,"num_object_copies":20528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3027}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6627","reported_seq":6763,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.388855+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:13.388855+0000","last_peered":"2024-04-26T02:29:13.388855+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:13.388855+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6627,"log_dups_size":0,"ondisk_log_size":6627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41878552,"num_objects":5113,"num_object_clones":0,"num_object_copies":20452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6546","reported_seq":6660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.246989+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:14.246989+0000","last_peered":"2024-04-26T02:29:14.246989+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:14.246989+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41103884,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2996}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6642","reported_seq":6751,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.314182+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:13.314182+0000","last_peered":"2024-04-26T02:29:13.314182+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:13.314182+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41585426,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6566","reported_seq":6676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.386854+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:13.386854+0000","last_peered":"2024-04-26T02:29:13.386854+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:13.386854+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41212166,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6706","reported_seq":6846,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:14.346214+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:14.346214+0000","last_peered":"2024-04-26T02:29:14.346214+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:14.346214+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41556230,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6833","reported_seq":6939,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:13.319898+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:13.319898+0000","last_peered":"2024-04-26T02:29:13.319898+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:13.319898+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43323916,"num_objects":5289,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6791","reported_seq":10996,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:17.827616+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:17.827616+0000","last_peered":"2024-04-26T02:29:17.827616+0000","last_clean":"2024-04-26T02:29:17.827616+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:17.827616+0000","last_undegraded":"2024-04-26T02:29:17.827616+0000","last_fullsized":"2024-04-26T02:29:17.827616+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6791,"log_dups_size":0,"ondisk_log_size":6791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:20.274371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41480716,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6709","reported_seq":6843,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:16.146862+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:16.146862+0000","last_peered":"2024-04-26T02:29:16.146862+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:16.146862+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41677324,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":663796928,"num_objects":81037,"num_object_clones":0,"num_object_copies":324148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106520,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4871,"num_bytes_recovered":36421632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1255776256,"data_stored":1255776256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106547,"ondisk_log_size":106547,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722748,"kb_used_data":325940,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93649092,"statfs":{"total":96636764160,"available":95896670208,"internally_reserved":0,"allocated":333762560,"data_stored":333568205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717488,"kb_used_data":325352,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93654352,"statfs":{"total":96636764160,"available":95902056448,"internally_reserved":0,"allocated":333160448,"data_stored":332966093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546932,"kb_used_data":254316,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93824908,"statfs":{"total":96636764160,"available":96076705792,"internally_reserved":0,"allocated":260419584,"data_stored":260225229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720032,"kb_used_data":323280,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93651808,"statfs":{"total":96636764160,"available":95899451392,"internally_reserved":0,"allocated":331038720,"data_stored":330844365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,5,6,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":330452992,"data_stored":330452992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260022272,"data_stored":260022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332173312,"data_stored":332173312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":333127680,"data_stored":333127680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:22.661 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:23.857 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:23.857 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:23.999 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-04-26T02:29:22.560862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654867648,"num_objects":79947,"num_object_clones":0,"num_object_copies":319788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107610,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4979,"num_bytes_recovered":37044224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107637,"ondisk_log_size":107637,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697520,"kb_used_data":1215028,"kb_used_omap":14,"kb_used_meta":1482417,"kb_avail":374789840,"statfs":{"total":386547056640,"available":383784796160,"internally_reserved":0,"allocated":1244188672,"data_stored":1243411252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1517995571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,17,6,13,6,1,0,0,3,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22061056,"num_objects":-2693,"num_object_clones":0,"num_object_copies":-10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003537"},"pg_stats":[{"pgid":"1.f","version":"36'6596","reported_seq":6695,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.484691+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:18.484691+0000","last_peered":"2024-04-26T02:29:18.484691+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:18.484691+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40370176,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6793","reported_seq":6920,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.193991+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:21.193991+0000","last_peered":"2024-04-26T02:29:21.193991+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:21.193991+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41275916,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6652","reported_seq":6778,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:19.304597+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:19.304597+0000","last_peered":"2024-04-26T02:29:19.304597+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:19.304597+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40778014,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6651,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2976}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6608","reported_seq":7491,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.250532+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:21.250532+0000","last_peered":"2024-04-26T02:29:21.250532+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:21.250532+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39907852,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6618","reported_seq":6743,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.336038+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:18.336038+0000","last_peered":"2024-04-26T02:29:18.336038+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:18.336038+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40784396,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6696","reported_seq":6835,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.420828+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:18.420828+0000","last_peered":"2024-04-26T02:29:18.420828+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:18.420828+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6785","reported_seq":11032,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:21.142213+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:21.142213+0000","last_peered":"2024-04-26T02:29:21.142213+0000","last_clean":"2024-04-26T02:29:21.142213+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:21.142213+0000","last_undegraded":"2024-04-26T02:29:21.142213+0000","last_fullsized":"2024-04-26T02:29:21.142213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:25.029241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735244,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6935","reported_seq":7092,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.273160+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:21.273160+0000","last_peered":"2024-04-26T02:29:21.273160+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:21.273160+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6935,"log_dups_size":0,"ondisk_log_size":6935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41398796,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6933,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6710","reported_seq":6848,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.460832+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:18.460832+0000","last_peered":"2024-04-26T02:29:18.460832+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:18.460832+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41198616,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6616","reported_seq":6732,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:19.334078+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:19.334078+0000","last_peered":"2024-04-26T02:29:19.334078+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:19.334078+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6616,"log_dups_size":0,"ondisk_log_size":6616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40530444,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6614,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2927}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6726","reported_seq":6837,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.553334+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:18.553334+0000","last_peered":"2024-04-26T02:29:18.553334+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:18.553334+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40897298,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6641","reported_seq":6757,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.490114+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:18.490114+0000","last_peered":"2024-04-26T02:29:18.490114+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:18.490114+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40597766,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6778","reported_seq":6930,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:19.376748+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:19.376748+0000","last_peered":"2024-04-26T02:29:19.376748+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:19.376748+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40966406,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6911","reported_seq":7018,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:18.430211+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:18.430211+0000","last_peered":"2024-04-26T02:29:18.430211+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:18.430211+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6911,"log_dups_size":0,"ondisk_log_size":6911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42684940,"num_objects":5211,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6791","reported_seq":10996,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:17.827616+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:17.827616+0000","last_peered":"2024-04-26T02:29:17.827616+0000","last_clean":"2024-04-26T02:29:17.827616+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:17.827616+0000","last_undegraded":"2024-04-26T02:29:17.827616+0000","last_fullsized":"2024-04-26T02:29:17.827616+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6791,"log_dups_size":0,"ondisk_log_size":6791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:20.274371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41480716,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6781","reported_seq":6921,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.381191+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:21.381191+0000","last_peered":"2024-04-26T02:29:21.381191+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:21.381191+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41087500,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654867648,"num_objects":79947,"num_object_clones":0,"num_object_copies":319788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107610,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4979,"num_bytes_recovered":37044224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1241444352,"data_stored":1241444352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107637,"ondisk_log_size":107637,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709860,"kb_used_data":321344,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93661980,"statfs":{"total":96636764160,"available":95909867520,"internally_reserved":0,"allocated":329056256,"data_stored":328861901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,12,1,3,2,1,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721152,"kb_used_data":320696,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93650688,"statfs":{"total":96636764160,"available":95898304512,"internally_reserved":0,"allocated":328392704,"data_stored":328198349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546932,"kb_used_data":254316,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93824908,"statfs":{"total":96636764160,"available":96076705792,"internally_reserved":0,"allocated":260419584,"data_stored":260225229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719576,"kb_used_data":318672,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93652264,"statfs":{"total":96636764160,"available":95899918336,"internally_reserved":0,"allocated":326320128,"data_stored":326125773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,6,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325738496,"data_stored":325738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260022272,"data_stored":260022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327434240,"data_stored":327434240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328249344,"data_stored":328249344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:24.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:25.453 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:25.454 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-26T02:29:24.561889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648314048,"num_objects":79147,"num_object_clones":0,"num_object_copies":316588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108410,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4997,"num_bytes_recovered":37044224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108437,"ondisk_log_size":108437,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2671192,"kb_used_data":1205344,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374816168,"statfs":{"total":386547056640,"available":383811756032,"internally_reserved":0,"allocated":1234272256,"data_stored":1233494836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1500956211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,0,4,12,14,5,2,0,1,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25681920,"num_objects":-3135,"num_object_clones":0,"num_object_copies":-12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004053"},"pg_stats":[{"pgid":"1.f","version":"36'6670","reported_seq":6773,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.643453+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:23.643453+0000","last_peered":"2024-04-26T02:29:23.643453+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:23.643453+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39763968,"num_objects":4854,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6793","reported_seq":6920,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.193991+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:21.193991+0000","last_peered":"2024-04-26T02:29:21.193991+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:21.193991+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41275916,"num_objects":5039,"num_object_clones":0,"num_object_copies":20156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6719","reported_seq":6845,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.355671+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:24.355671+0000","last_peered":"2024-04-26T02:29:24.355671+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:24.355671+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40229150,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6718,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6608","reported_seq":7491,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.250532+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:21.250532+0000","last_peered":"2024-04-26T02:29:21.250532+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:21.250532+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39907852,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6606,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6698","reported_seq":6831,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.635981+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:23.635981+0000","last_peered":"2024-04-26T02:29:23.635981+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:23.635981+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40129036,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6765","reported_seq":6905,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.613780+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:23.613780+0000","last_peered":"2024-04-26T02:29:23.613780+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:23.613780+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6765,"log_dups_size":0,"ondisk_log_size":6765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39608320,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6785","reported_seq":11032,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:21.142213+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:21.142213+0000","last_peered":"2024-04-26T02:29:21.142213+0000","last_clean":"2024-04-26T02:29:21.142213+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:21.142213+0000","last_undegraded":"2024-04-26T02:29:21.142213+0000","last_fullsized":"2024-04-26T02:29:21.142213+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:25.029241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40735244,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'6935","reported_seq":7092,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.273160+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:21.273160+0000","last_peered":"2024-04-26T02:29:21.273160+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:21.273160+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6935,"log_dups_size":0,"ondisk_log_size":6935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41398796,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6933,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6784","reported_seq":6926,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.608759+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:23.608759+0000","last_peered":"2024-04-26T02:29:23.608759+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:23.608759+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6784,"log_dups_size":0,"ondisk_log_size":6784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40592408,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6683","reported_seq":6799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.229114+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:24.229114+0000","last_peered":"2024-04-26T02:29:24.229114+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:24.229114+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39981580,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6681,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2860}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6796","reported_seq":6913,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.568255+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:23.568255+0000","last_peered":"2024-04-26T02:29:23.568255+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:23.568255+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6796,"log_dups_size":0,"ondisk_log_size":6796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40323858,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6707","reported_seq":6823,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.450175+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:23.450175+0000","last_peered":"2024-04-26T02:29:23.450175+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:23.450175+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40057094,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6848","reported_seq":7008,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.073817+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:24.073817+0000","last_peered":"2024-04-26T02:29:24.073817+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:24.073817+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40392966,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6992","reported_seq":7106,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.576261+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:23.576261+0000","last_peered":"2024-04-26T02:29:23.576261+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:23.576261+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42021388,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6990,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3087}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6873","reported_seq":11078,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:22.837272+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:22.837272+0000","last_peered":"2024-04-26T02:29:22.837272+0000","last_clean":"2024-04-26T02:29:22.837272+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:22.837272+0000","last_undegraded":"2024-04-26T02:29:22.837272+0000","last_fullsized":"2024-04-26T02:29:22.837272+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:25.294551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808972,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6781","reported_seq":6921,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:21.381191+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:21.381191+0000","last_peered":"2024-04-26T02:29:21.381191+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:21.381191+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41087500,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":648314048,"num_objects":79147,"num_object_clones":0,"num_object_copies":316588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108410,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4997,"num_bytes_recovered":37044224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231351808,"data_stored":1231351808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108437,"ondisk_log_size":108437,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709504,"kb_used_data":316856,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93662336,"statfs":{"total":96636764160,"available":95910232064,"internally_reserved":0,"allocated":324460544,"data_stored":324266189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,0,4,5,1,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":316072,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":323657728,"data_stored":323463373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,0,1,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538048,"kb_used_data":253744,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93833792,"statfs":{"total":96636764160,"available":96085803008,"internally_reserved":0,"allocated":259833856,"data_stored":259639501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719576,"kb_used_data":318672,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93652264,"statfs":{"total":96636764160,"available":95899918336,"internally_reserved":0,"allocated":326320128,"data_stored":326125773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,6,2,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":325738496,"data_stored":325738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259465216,"data_stored":259465216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322707456,"data_stored":322707456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323440640,"data_stored":323440640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6670", 'reported_seq': 6773, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.643453+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:23.643453+0000', 'last_peered': '2024-04-26T02:29:23.643453+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:23.643453+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6670, 'log_dups_size': 0, 'ondisk_log_size': 6670, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39763968, 'num_objects': 4854, 'num_object_clones': 0, 'num_object_copies': 19416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6670, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2865}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'6793", 'reported_seq': 6920, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:21.193991+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:21.193991+0000', 'last_peered': '2024-04-26T02:29:21.193991+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:21.193991+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6793, 'log_dups_size': 0, 'ondisk_log_size': 6793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41275916, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 20156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6791, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6719", 'reported_seq': 6845, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:24.355671+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:24.355671+0000', 'last_peered': '2024-04-26T02:29:24.355671+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:24.355671+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6719, 'log_dups_size': 0, 'ondisk_log_size': 6719, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40229150, 'num_objects': 4912, 'num_object_clones': 0, 'num_object_copies': 19648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4912, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6718, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2909}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:25.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6608", 'reported_seq': 7491, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:21.250532+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:21.250532+0000', 'last_peered': '2024-04-26T02:29:21.250532+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:21.250532+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6608, 'log_dups_size': 0, 'ondisk_log_size': 6608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39907852, 'num_objects': 4872, 'num_object_clones': 0, 'num_object_copies': 19488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1563, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6606, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 409, 'num_bytes_recovered': 3170304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6698", 'reported_seq': 6831, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.635981+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:23.635981+0000', 'last_peered': '2024-04-26T02:29:23.635981+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:23.635981+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6698, 'log_dups_size': 0, 'ondisk_log_size': 6698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40129036, 'num_objects': 4899, 'num_object_clones': 0, 'num_object_copies': 19596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4899, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6696, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2924}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6765", 'reported_seq': 6905, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.613780+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:23.613780+0000', 'last_peered': '2024-04-26T02:29:23.613780+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:23.613780+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6765, 'log_dups_size': 0, 'ondisk_log_size': 6765, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39608320, 'num_objects': 4835, 'num_object_clones': 0, 'num_object_copies': 19340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6765, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2830}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'6935", 'reported_seq': 7092, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:21.273160+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:21.273160+0000', 'last_peered': '2024-04-26T02:29:21.273160+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:21.273160+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6935, 'log_dups_size': 0, 'ondisk_log_size': 6935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41398796, 'num_objects': 5054, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6933, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:25.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6784", 'reported_seq': 6926, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.608759+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:23.608759+0000', 'last_peered': '2024-04-26T02:29:23.608759+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:23.608759+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6784, 'log_dups_size': 0, 'ondisk_log_size': 6784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40592408, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1967, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6781, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6683", 'reported_seq': 6799, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:24.229114+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:24.229114+0000', 'last_peered': '2024-04-26T02:29:24.229114+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:24.229114+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6683, 'log_dups_size': 0, 'ondisk_log_size': 6683, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39981580, 'num_objects': 4881, 'num_object_clones': 0, 'num_object_copies': 19524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2023, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6681, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2860}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6796", 'reported_seq': 6913, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.568255+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:23.568255+0000', 'last_peered': '2024-04-26T02:29:23.568255+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:23.568255+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6796, 'log_dups_size': 0, 'ondisk_log_size': 6796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40323858, 'num_objects': 4923, 'num_object_clones': 0, 'num_object_copies': 19692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6793, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6707", 'reported_seq': 6823, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.450175+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:23.450175+0000', 'last_peered': '2024-04-26T02:29:23.450175+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:23.450175+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6707, 'log_dups_size': 0, 'ondisk_log_size': 6707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40057094, 'num_objects': 4890, 'num_object_clones': 0, 'num_object_copies': 19560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6706, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:25.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'6848", 'reported_seq': 7008, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:24.073817+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:24.073817+0000', 'last_peered': '2024-04-26T02:29:24.073817+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:24.073817+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6848, 'log_dups_size': 0, 'ondisk_log_size': 6848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40392966, 'num_objects': 4931, 'num_object_clones': 0, 'num_object_copies': 19724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6847, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:25.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:25.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'6992", 'reported_seq': 7106, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:23.576261+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:23.576261+0000', 'last_peered': '2024-04-26T02:29:23.576261+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:23.576261+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6992, 'log_dups_size': 0, 'ondisk_log_size': 6992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42021388, 'num_objects': 5130, 'num_object_clones': 0, 'num_object_copies': 20520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6990, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:25.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:25.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'6781", 'reported_seq': 6921, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:21.381191+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:21.381191+0000', 'last_peered': '2024-04-26T02:29:21.381191+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:21.381191+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6781, 'log_dups_size': 0, 'ondisk_log_size': 6781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41087500, 'num_objects': 5016, 'num_object_clones': 0, 'num_object_copies': 20064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6779, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2987}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:25.522 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:27.060 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:27.060 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27697/315068 objects degraded (8.791%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78767,"data_bytes":645201088,"bytes_used":2721939456,"bytes_avail":383825117184,"bytes_total":386547056640,"degraded_objects":27697,"degraded_total":315068,"degraded_ratio":0.087908007160359028,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":141264,"recovering_keys_per_sec":0,"num_objects_recovered":284,"num_bytes_recovered":1695744,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:27.215 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:27.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:28.424 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:28.424 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:28.486 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-26T02:29:26.562415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645201088,"num_objects":78767,"num_object_clones":0,"num_object_copies":315068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108790,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5072,"num_bytes_recovered":37601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108817,"ondisk_log_size":108817,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658144,"kb_used_data":1200616,"kb_used_omap":14,"kb_used_meta":1457457,"kb_avail":374829216,"statfs":{"total":386547056640,"available":383825117184,"internally_reserved":0,"allocated":1229430784,"data_stored":1228653364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1492436531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,0,3,11,9,5,2,0,1,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22233088,"num_objects":-2714,"num_object_clones":0,"num_object_copies":-10856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004071"},"pg_stats":[{"pgid":"1.f","version":"36'6670","reported_seq":6773,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.643453+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:23.643453+0000","last_peered":"2024-04-26T02:29:23.643453+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:23.643453+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39763968,"num_objects":4854,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6876","reported_seq":7005,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:26.403242+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:26.403242+0000","last_peered":"2024-04-26T02:29:26.403242+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:26.403242+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40595980,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6719","reported_seq":6845,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.355671+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:24.355671+0000","last_peered":"2024-04-26T02:29:24.355671+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:24.355671+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40229150,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6718,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6682","reported_seq":7701,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:26.396574+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:26.396574+0000","last_peered":"2024-04-26T02:29:26.396574+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:26.396574+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39301644,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":3727360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6698","reported_seq":6831,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.635981+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:23.635981+0000","last_peered":"2024-04-26T02:29:23.635981+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:23.635981+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40129036,"num_objects":4899,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6765","reported_seq":6905,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.613780+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:23.613780+0000","last_peered":"2024-04-26T02:29:23.613780+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:23.613780+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6765,"log_dups_size":0,"ondisk_log_size":6765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39608320,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6860","reported_seq":11107,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:26.316283+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:26.316283+0000","last_peered":"2024-04-26T02:29:26.316283+0000","last_clean":"2024-04-26T02:29:26.316283+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:26.316283+0000","last_undegraded":"2024-04-26T02:29:26.316283+0000","last_fullsized":"2024-04-26T02:29:26.316283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6860,"log_dups_size":0,"ondisk_log_size":6860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:30.110744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40120844,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6858,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7000","reported_seq":7163,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:26.349718+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:26.349718+0000","last_peered":"2024-04-26T02:29:26.349718+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:26.349718+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7000,"log_dups_size":0,"ondisk_log_size":7000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40866316,"num_objects":4989,"num_object_clones":0,"num_object_copies":19956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6998,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6784","reported_seq":6926,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.608759+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:23.608759+0000","last_peered":"2024-04-26T02:29:23.608759+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:23.608759+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6784,"log_dups_size":0,"ondisk_log_size":6784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40592408,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6683","reported_seq":6799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.229114+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:24.229114+0000","last_peered":"2024-04-26T02:29:24.229114+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:24.229114+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39981580,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6681,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2860}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6796","reported_seq":6913,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.568255+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:23.568255+0000","last_peered":"2024-04-26T02:29:23.568255+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:23.568255+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6796,"log_dups_size":0,"ondisk_log_size":6796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40323858,"num_objects":4923,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6707","reported_seq":6823,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.450175+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:23.450175+0000","last_peered":"2024-04-26T02:29:23.450175+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:23.450175+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40057094,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6848","reported_seq":7008,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:24.073817+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:24.073817+0000","last_peered":"2024-04-26T02:29:24.073817+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:24.073817+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40392966,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'6992","reported_seq":7106,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:23.576261+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:23.576261+0000","last_peered":"2024-04-26T02:29:23.576261+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:23.576261+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42021388,"num_objects":5130,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6990,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3087}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6873","reported_seq":11078,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:22.837272+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:22.837272+0000","last_peered":"2024-04-26T02:29:22.837272+0000","last_clean":"2024-04-26T02:29:22.837272+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:22.837272+0000","last_undegraded":"2024-04-26T02:29:22.837272+0000","last_fullsized":"2024-04-26T02:29:22.837272+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:25.294551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808972,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6864","reported_seq":7010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:26.162011+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:26.162011+0000","last_peered":"2024-04-26T02:29:26.162011+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:26.162011+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40407564,"num_objects":4933,"num_object_clones":0,"num_object_copies":19732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6862,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645201088,"num_objects":78767,"num_object_clones":0,"num_object_copies":315068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108790,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5072,"num_bytes_recovered":37601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226420224,"data_stored":1226420224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108817,"ondisk_log_size":108817,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709504,"kb_used_data":316856,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93662336,"statfs":{"total":96636764160,"available":95910232064,"internally_reserved":0,"allocated":324460544,"data_stored":324266189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,0,4,5,1,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704064,"kb_used_data":316072,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93667776,"statfs":{"total":96636764160,"available":95915802624,"internally_reserved":0,"allocated":323657728,"data_stored":323463373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,0,1,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538048,"kb_used_data":253744,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93833792,"statfs":{"total":96636764160,"available":96085803008,"internally_reserved":0,"allocated":259833856,"data_stored":259639501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706528,"kb_used_data":313944,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93665312,"statfs":{"total":96636764160,"available":95913279488,"internally_reserved":0,"allocated":321478656,"data_stored":321284301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":401993423},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320806912,"data_stored":320806912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259465216,"data_stored":259465216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322707456,"data_stored":322707456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323440640,"data_stored":323440640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:31.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:32.606 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:32.606 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:32.685 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-04-26T02:29:32.565328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635919552,"num_objects":77634,"num_object_clones":0,"num_object_copies":310536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109923,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5202,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109950,"ondisk_log_size":109950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660928,"kb_used_data":1186784,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374826432,"statfs":{"total":386547056640,"available":383822266368,"internally_reserved":0,"allocated":1215266816,"data_stored":1214489396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1509475891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,6,17,6,6,3,0,2,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21839872,"num_objects":-2666,"num_object_clones":0,"num_object_copies":-10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":2097152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005028"},"pg_stats":[{"pgid":"1.f","version":"36'6742","reported_seq":6847,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.543996+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:28.543996+0000","last_peered":"2024-04-26T02:29:28.543996+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:28.543996+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39174144,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6947","reported_seq":7076,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.598705+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:31.598705+0000","last_peered":"2024-04-26T02:29:31.598705+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:31.598705+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6947,"log_dups_size":0,"ondisk_log_size":6947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40014348,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6794","reported_seq":6920,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.385620+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:29.385620+0000","last_peered":"2024-04-26T02:29:29.385620+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:29.385620+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6794,"log_dups_size":0,"ondisk_log_size":6794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39614750,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6793,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6750","reported_seq":7993,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.520341+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:31.520341+0000","last_peered":"2024-04-26T02:29:31.520341+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:31.520341+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38744588,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":4644864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6776","reported_seq":6917,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.507031+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:28.507031+0000","last_peered":"2024-04-26T02:29:28.507031+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:28.507031+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39490060,"num_objects":4821,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6774,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6835","reported_seq":6978,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.586731+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:28.586731+0000","last_peered":"2024-04-26T02:29:28.586731+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:28.586731+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6835,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6941","reported_seq":11188,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:31.454469+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:31.454469+0000","last_peered":"2024-04-26T02:29:31.454469+0000","last_clean":"2024-04-26T02:29:31.454469+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:31.454469+0000","last_undegraded":"2024-04-26T02:29:31.454469+0000","last_fullsized":"2024-04-26T02:29:31.454469+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:35.961522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39457292,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6939,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7061","reported_seq":7226,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.540498+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:31.540498+0000","last_peered":"2024-04-26T02:29:31.540498+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:31.540498+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40366604,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6849","reported_seq":6993,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.687964+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:28.687964+0000","last_peered":"2024-04-26T02:29:28.687964+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:28.687964+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40059928,"num_objects":4891,"num_object_clones":0,"num_object_copies":19564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6762","reported_seq":6878,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.401133+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:29.401133+0000","last_peered":"2024-04-26T02:29:29.401133+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:29.401133+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39334412,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6760,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6857","reported_seq":6978,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.314243+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:28.314243+0000","last_peered":"2024-04-26T02:29:28.314243+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:28.314243+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39824146,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6770","reported_seq":6890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.619428+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:28.619428+0000","last_peered":"2024-04-26T02:29:28.619428+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:28.619428+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39540998,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6920","reported_seq":7082,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.362612+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:29.362612+0000","last_peered":"2024-04-26T02:29:29.362612+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:29.362612+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39803142,"num_objects":4859,"num_object_clones":0,"num_object_copies":19436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6919,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7052","reported_seq":7174,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.543684+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:28.543684+0000","last_peered":"2024-04-26T02:29:28.543684+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:28.543684+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41529868,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7050,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3031}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6951","reported_seq":11156,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:27.970448+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:27.970448+0000","last_peered":"2024-04-26T02:29:27.970448+0000","last_clean":"2024-04-26T02:29:27.970448+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:27.970448+0000","last_undegraded":"2024-04-26T02:29:27.970448+0000","last_fullsized":"2024-04-26T02:29:27.970448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:31.246946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40169996,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6943","reported_seq":7090,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.535079+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:31.535079+0000","last_peered":"2024-04-26T02:29:31.535079+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:31.535079+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39760396,"num_objects":4854,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6941,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635919552,"num_objects":77634,"num_object_clones":0,"num_object_copies":310536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109923,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5202,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212018688,"data_stored":1212018688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109950,"ondisk_log_size":109950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708920,"kb_used_data":312112,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93662920,"statfs":{"total":96636764160,"available":95910830080,"internally_reserved":0,"allocated":319602688,"data_stored":319408333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,9,4,1,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703608,"kb_used_data":311472,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93668232,"statfs":{"total":96636764160,"available":95916269568,"internally_reserved":0,"allocated":318947328,"data_stored":318752973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541640,"kb_used_data":253184,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93830200,"statfs":{"total":96636764160,"available":96082124800,"internally_reserved":0,"allocated":259260416,"data_stored":259066061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706760,"kb_used_data":310016,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93665080,"statfs":{"total":96636764160,"available":95913041920,"internally_reserved":0,"allocated":317456384,"data_stored":317262029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,2,2,5,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316465152,"data_stored":316465152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258842624,"data_stored":258842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317997056,"data_stored":317997056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318713856,"data_stored":318713856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:32.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:34.057 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:34.057 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:34.169 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-04-26T02:29:32.565328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635919552,"num_objects":77634,"num_object_clones":0,"num_object_copies":310536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109923,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5202,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109950,"ondisk_log_size":109950,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660928,"kb_used_data":1186784,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374826432,"statfs":{"total":386547056640,"available":383822266368,"internally_reserved":0,"allocated":1215266816,"data_stored":1214489396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1509475891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,6,17,6,6,3,0,2,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21839872,"num_objects":-2666,"num_object_clones":0,"num_object_copies":-10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":2097152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005028"},"pg_stats":[{"pgid":"1.f","version":"36'6742","reported_seq":6847,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.543996+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:28.543996+0000","last_peered":"2024-04-26T02:29:28.543996+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:28.543996+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39174144,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6947","reported_seq":7076,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.598705+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:31.598705+0000","last_peered":"2024-04-26T02:29:31.598705+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:31.598705+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6947,"log_dups_size":0,"ondisk_log_size":6947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40014348,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6794","reported_seq":6920,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.385620+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:29.385620+0000","last_peered":"2024-04-26T02:29:29.385620+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:29.385620+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6794,"log_dups_size":0,"ondisk_log_size":6794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39614750,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6793,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6750","reported_seq":7993,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.520341+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:31.520341+0000","last_peered":"2024-04-26T02:29:31.520341+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:31.520341+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38744588,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":4644864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6776","reported_seq":6917,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.507031+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:28.507031+0000","last_peered":"2024-04-26T02:29:28.507031+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:28.507031+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39490060,"num_objects":4821,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6774,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6835","reported_seq":6978,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.586731+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:28.586731+0000","last_peered":"2024-04-26T02:29:28.586731+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:28.586731+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6835,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6941","reported_seq":11188,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:31.454469+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:31.454469+0000","last_peered":"2024-04-26T02:29:31.454469+0000","last_clean":"2024-04-26T02:29:31.454469+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:31.454469+0000","last_undegraded":"2024-04-26T02:29:31.454469+0000","last_fullsized":"2024-04-26T02:29:31.454469+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:35.961522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39457292,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6939,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7061","reported_seq":7226,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.540498+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:31.540498+0000","last_peered":"2024-04-26T02:29:31.540498+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:31.540498+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40366604,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6849","reported_seq":6993,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.687964+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:28.687964+0000","last_peered":"2024-04-26T02:29:28.687964+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:28.687964+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40059928,"num_objects":4891,"num_object_clones":0,"num_object_copies":19564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6762","reported_seq":6878,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.401133+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:29.401133+0000","last_peered":"2024-04-26T02:29:29.401133+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:29.401133+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39334412,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6760,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6857","reported_seq":6978,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.314243+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:28.314243+0000","last_peered":"2024-04-26T02:29:28.314243+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:28.314243+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39824146,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6770","reported_seq":6890,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.619428+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:28.619428+0000","last_peered":"2024-04-26T02:29:28.619428+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:28.619428+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39540998,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2877}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6920","reported_seq":7082,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:29.362612+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:29.362612+0000","last_peered":"2024-04-26T02:29:29.362612+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:29.362612+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39803142,"num_objects":4859,"num_object_clones":0,"num_object_copies":19436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6919,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7052","reported_seq":7174,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:28.543684+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:28.543684+0000","last_peered":"2024-04-26T02:29:28.543684+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:28.543684+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41529868,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7050,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3031}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'6951","reported_seq":11156,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:27.970448+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:27.970448+0000","last_peered":"2024-04-26T02:29:27.970448+0000","last_clean":"2024-04-26T02:29:27.970448+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:27.970448+0000","last_undegraded":"2024-04-26T02:29:27.970448+0000","last_fullsized":"2024-04-26T02:29:27.970448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:31.246946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40169996,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6943","reported_seq":7090,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.535079+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:31.535079+0000","last_peered":"2024-04-26T02:29:31.535079+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:31.535079+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39760396,"num_objects":4854,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6941,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635919552,"num_objects":77634,"num_object_clones":0,"num_object_copies":310536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109923,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5202,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212018688,"data_stored":1212018688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109950,"ondisk_log_size":109950,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708920,"kb_used_data":312112,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93662920,"statfs":{"total":96636764160,"available":95910830080,"internally_reserved":0,"allocated":319602688,"data_stored":319408333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,9,4,1,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703608,"kb_used_data":311472,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93668232,"statfs":{"total":96636764160,"available":95916269568,"internally_reserved":0,"allocated":318947328,"data_stored":318752973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541640,"kb_used_data":253184,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93830200,"statfs":{"total":96636764160,"available":96082124800,"internally_reserved":0,"allocated":259260416,"data_stored":259066061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706760,"kb_used_data":310016,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93665080,"statfs":{"total":96636764160,"available":95913041920,"internally_reserved":0,"allocated":317456384,"data_stored":317262029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,2,2,5,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316465152,"data_stored":316465152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258842624,"data_stored":258842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317997056,"data_stored":317997056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":318713856,"data_stored":318713856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6742", 'reported_seq': 6847, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.543996+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:28.543996+0000', 'last_peered': '2024-04-26T02:29:28.543996+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:28.543996+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6742, 'log_dups_size': 0, 'ondisk_log_size': 6742, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39174144, 'num_objects': 4782, 'num_object_clones': 0, 'num_object_copies': 19128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1988, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6742, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2794}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'6947", 'reported_seq': 7076, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:31.598705+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:31.598705+0000', 'last_peered': '2024-04-26T02:29:31.598705+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:31.598705+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6947, 'log_dups_size': 0, 'ondisk_log_size': 6947, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40014348, 'num_objects': 4885, 'num_object_clones': 0, 'num_object_copies': 19540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6945, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6794", 'reported_seq': 6920, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:29.385620+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:29.385620+0000', 'last_peered': '2024-04-26T02:29:29.385620+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:29.385620+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6794, 'log_dups_size': 0, 'ondisk_log_size': 6794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39614750, 'num_objects': 4837, 'num_object_clones': 0, 'num_object_copies': 19348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4837, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6793, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2834}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:34.170 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6750", 'reported_seq': 7993, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:31.520341+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:31.520341+0000', 'last_peered': '2024-04-26T02:29:31.520341+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:31.520341+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6750, 'log_dups_size': 0, 'ondisk_log_size': 6750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38744588, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1383, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6748, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 589, 'num_bytes_recovered': 4644864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6776", 'reported_seq': 6917, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.507031+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:28.507031+0000', 'last_peered': '2024-04-26T02:29:28.507031+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:28.507031+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6776, 'log_dups_size': 0, 'ondisk_log_size': 6776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39490060, 'num_objects': 4821, 'num_object_clones': 0, 'num_object_copies': 19284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6774, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2850}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6835", 'reported_seq': 6978, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.586731+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:28.586731+0000', 'last_peered': '2024-04-26T02:29:28.586731+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:28.586731+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6835, 'log_dups_size': 0, 'ondisk_log_size': 6835, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39034880, 'num_objects': 4765, 'num_object_clones': 0, 'num_object_copies': 19060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2004, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6835, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7061", 'reported_seq': 7226, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:31.540498+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:31.540498+0000', 'last_peered': '2024-04-26T02:29:31.540498+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:31.540498+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7061, 'log_dups_size': 0, 'ondisk_log_size': 7061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40366604, 'num_objects': 4928, 'num_object_clones': 0, 'num_object_copies': 19712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7059, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6849", 'reported_seq': 6993, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.687964+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:28.687964+0000', 'last_peered': '2024-04-26T02:29:28.687964+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:28.687964+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6849, 'log_dups_size': 0, 'ondisk_log_size': 6849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40059928, 'num_objects': 4891, 'num_object_clones': 0, 'num_object_copies': 19564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6846, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2928}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:34.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6762", 'reported_seq': 6878, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:29.401133+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:29.401133+0000', 'last_peered': '2024-04-26T02:29:29.401133+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:29.401133+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6762, 'log_dups_size': 0, 'ondisk_log_size': 6762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39334412, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2023, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6760, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2781}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6857", 'reported_seq': 6978, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.314243+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:28.314243+0000', 'last_peered': '2024-04-26T02:29:28.314243+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:28.314243+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6857, 'log_dups_size': 0, 'ondisk_log_size': 6857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39824146, 'num_objects': 4862, 'num_object_clones': 0, 'num_object_copies': 19448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6854, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2874}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6770", 'reported_seq': 6890, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.619428+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:28.619428+0000', 'last_peered': '2024-04-26T02:29:28.619428+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:28.619428+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6770, 'log_dups_size': 0, 'ondisk_log_size': 6770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39540998, 'num_objects': 4827, 'num_object_clones': 0, 'num_object_copies': 19308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1951, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6769, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2877}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'6920", 'reported_seq': 7082, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:29.362612+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:29.362612+0000', 'last_peered': '2024-04-26T02:29:29.362612+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:29.362612+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6920, 'log_dups_size': 0, 'ondisk_log_size': 6920, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39803142, 'num_objects': 4859, 'num_object_clones': 0, 'num_object_copies': 19436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6919, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7052", 'reported_seq': 7174, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:28.543684+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:28.543684+0000', 'last_peered': '2024-04-26T02:29:28.543684+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:28.543684+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7052, 'log_dups_size': 0, 'ondisk_log_size': 7052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41529868, 'num_objects': 5070, 'num_object_clones': 0, 'num_object_copies': 20280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7050, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:34.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'6943", 'reported_seq': 7090, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:31.535079+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:31.535079+0000', 'last_peered': '2024-04-26T02:29:31.535079+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:31.535079+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6943, 'log_dups_size': 0, 'ondisk_log_size': 6943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39760396, 'num_objects': 4854, 'num_object_clones': 0, 'num_object_copies': 19416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2027, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6941, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:34.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:35.401 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:35.401 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27540/307792 objects degraded (8.948%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76948,"data_bytes":630299840,"bytes_used":2737352704,"bytes_avail":383809703936,"bytes_total":386547056640,"degraded_objects":27540,"degraded_total":307792,"degraded_ratio":0.089476009772833598,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":122828,"recovering_keys_per_sec":0,"num_objects_recovered":250,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":249},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:35.420 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:35.420 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:36.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:36.261 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:36.319 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-26T02:29:34.565853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630299840,"num_objects":76948,"num_object_clones":0,"num_object_copies":307792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110609,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5229,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110636,"ondisk_log_size":110636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673196,"kb_used_data":1178244,"kb_used_omap":14,"kb_used_meta":1494897,"kb_avail":374814164,"statfs":{"total":386547056640,"available":383809703936,"internally_reserved":0,"allocated":1206521856,"data_stored":1205744436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1530775091},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,3,3,8,7,12,2,5,0,0,0,5],"upper_bound":8192},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24567808,"num_objects":-2999,"num_object_clones":0,"num_object_copies":-11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004991"},"pg_stats":[{"pgid":"1.f","version":"36'6808","reported_seq":6929,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.818900+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:33.818900+0000","last_peered":"2024-04-26T02:29:33.818900+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:33.818900+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38633472,"num_objects":4716,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'6947","reported_seq":7076,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.598705+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:31.598705+0000","last_peered":"2024-04-26T02:29:31.598705+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:31.598705+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6947,"log_dups_size":0,"ondisk_log_size":6947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40014348,"num_objects":4885,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6855","reported_seq":6987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.426047+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:34.426047+0000","last_peered":"2024-04-26T02:29:34.426047+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:34.426047+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39115038,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6854,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6750","reported_seq":7993,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.520341+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:31.520341+0000","last_peered":"2024-04-26T02:29:31.520341+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:31.520341+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38744588,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":4644864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6848","reported_seq":6989,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.775630+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:33.775630+0000","last_peered":"2024-04-26T02:29:33.775630+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:33.775630+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900236,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6897","reported_seq":7046,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.811343+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:33.811343+0000","last_peered":"2024-04-26T02:29:33.811343+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:33.811343+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38526976,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'6941","reported_seq":11188,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:31.454469+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:31.454469+0000","last_peered":"2024-04-26T02:29:31.454469+0000","last_clean":"2024-04-26T02:29:31.454469+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:31.454469+0000","last_undegraded":"2024-04-26T02:29:31.454469+0000","last_fullsized":"2024-04-26T02:29:31.454469+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:35.961522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39457292,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6939,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7061","reported_seq":7226,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.540498+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:31.540498+0000","last_peered":"2024-04-26T02:29:31.540498+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:31.540498+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40366604,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6913","reported_seq":7061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.758293+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:33.758293+0000","last_peered":"2024-04-26T02:29:33.758293+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:33.758293+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39535640,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6823","reported_seq":6941,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.390105+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:34.390105+0000","last_peered":"2024-04-26T02:29:34.390105+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:34.390105+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38834700,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6919","reported_seq":7048,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.797059+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:33.797059+0000","last_peered":"2024-04-26T02:29:33.797059+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:33.797059+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6919,"log_dups_size":0,"ondisk_log_size":6919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39316242,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6829","reported_seq":6953,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.715017+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:33.715017+0000","last_peered":"2024-04-26T02:29:33.715017+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:33.715017+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39057670,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6980","reported_seq":7147,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.435111+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:34.435111+0000","last_peered":"2024-04-26T02:29:34.435111+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:34.435111+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39311622,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7109","reported_seq":7233,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.729206+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:33.729206+0000","last_peered":"2024-04-26T02:29:33.729206+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:33.729206+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41062924,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7107,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7013","reported_seq":11218,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:32.748401+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:32.748401+0000","last_peered":"2024-04-26T02:29:32.748401+0000","last_clean":"2024-04-26T02:29:32.748401+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:32.748401+0000","last_undegraded":"2024-04-26T02:29:32.748401+0000","last_fullsized":"2024-04-26T02:29:32.748401+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7013,"log_dups_size":0,"ondisk_log_size":7013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:36.331194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39662092,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'6943","reported_seq":7090,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:31.535079+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:31.535079+0000","last_peered":"2024-04-26T02:29:31.535079+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:31.535079+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39760396,"num_objects":4854,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6941,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":630299840,"num_objects":76948,"num_object_clones":0,"num_object_copies":307792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110609,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5229,"num_bytes_recovered":38518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203318784,"data_stored":1203318784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110636,"ondisk_log_size":110636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":308064,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":315457536,"data_stored":315263181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,5,6,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707856,"kb_used_data":307392,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93663984,"statfs":{"total":96636764160,"available":95911919616,"internally_reserved":0,"allocated":314769408,"data_stored":314575053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,0,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545388,"kb_used_data":252772,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93826452,"statfs":{"total":96636764160,"available":96078286848,"internally_reserved":0,"allocated":258838528,"data_stored":258644173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706760,"kb_used_data":310016,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93665080,"statfs":{"total":96636764160,"available":95913041920,"internally_reserved":0,"allocated":317456384,"data_stored":317262029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,2,2,5,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316465152,"data_stored":316465152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258486272,"data_stored":258486272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313892864,"data_stored":313892864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314474496,"data_stored":314474496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:39.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:40.171 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:40.172 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:40.231 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-26T02:29:38.569089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626777280,"num_objects":76518,"num_object_clones":0,"num_object_copies":306072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111039,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5332,"num_bytes_recovered":39264256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111066,"ondisk_log_size":111066,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660020,"kb_used_data":1173388,"kb_used_omap":14,"kb_used_meta":1486577,"kb_avail":374827340,"statfs":{"total":386547056640,"available":383823196160,"internally_reserved":0,"allocated":1201549312,"data_stored":1200771892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1522255411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,8,9,8,10,2,7,0,0,2,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18423808,"num_objects":-2249,"num_object_clones":0,"num_object_copies":-8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006673"},"pg_stats":[{"pgid":"1.f","version":"36'6808","reported_seq":6929,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.818900+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:33.818900+0000","last_peered":"2024-04-26T02:29:33.818900+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:33.818900+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6808,"log_dups_size":0,"ondisk_log_size":6808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38633472,"num_objects":4716,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7003","reported_seq":7144,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.588385+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:36.588385+0000","last_peered":"2024-04-26T02:29:36.588385+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:36.588385+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555596,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6855","reported_seq":6987,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.426047+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:34.426047+0000","last_peered":"2024-04-26T02:29:34.426047+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:34.426047+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39115038,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6854,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6821","reported_seq":8245,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.689905+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:36.689905+0000","last_peered":"2024-04-26T02:29:36.689905+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:36.689905+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162956,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6848","reported_seq":6989,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.775630+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:33.775630+0000","last_peered":"2024-04-26T02:29:33.775630+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:33.775630+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900236,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6846,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6897","reported_seq":7046,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.811343+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:33.811343+0000","last_peered":"2024-04-26T02:29:33.811343+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:33.811343+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38526976,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7016","reported_seq":11263,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:36.455489+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:36.455489+0000","last_peered":"2024-04-26T02:29:36.455489+0000","last_clean":"2024-04-26T02:29:36.455489+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:36.455489+0000","last_undegraded":"2024-04-26T02:29:36.455489+0000","last_fullsized":"2024-04-26T02:29:36.455489+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:41.031765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842892,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7143","reported_seq":7314,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.619538+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:36.619538+0000","last_peered":"2024-04-26T02:29:36.619538+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:36.619538+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39694860,"num_objects":4846,"num_object_clones":0,"num_object_copies":19384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6913","reported_seq":7061,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.758293+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:33.758293+0000","last_peered":"2024-04-26T02:29:33.758293+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:33.758293+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39535640,"num_objects":4827,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6823","reported_seq":6941,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.390105+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:34.390105+0000","last_peered":"2024-04-26T02:29:34.390105+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:34.390105+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38834700,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6919","reported_seq":7048,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.797059+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:33.797059+0000","last_peered":"2024-04-26T02:29:33.797059+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:33.797059+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6919,"log_dups_size":0,"ondisk_log_size":6919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39316242,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6829","reported_seq":6953,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.715017+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:33.715017+0000","last_peered":"2024-04-26T02:29:33.715017+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:33.715017+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39057670,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'6980","reported_seq":7147,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:34.435111+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:34.435111+0000","last_peered":"2024-04-26T02:29:34.435111+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:34.435111+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39311622,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7109","reported_seq":7233,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:33.729206+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:33.729206+0000","last_peered":"2024-04-26T02:29:33.729206+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:33.729206+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41062924,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7107,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7087","reported_seq":11292,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:38.055470+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:38.055470+0000","last_peered":"2024-04-26T02:29:38.055470+0000","last_clean":"2024-04-26T02:29:38.055470+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:38.055470+0000","last_undegraded":"2024-04-26T02:29:38.055470+0000","last_fullsized":"2024-04-26T02:29:38.055470+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:42.306447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39055884,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7085,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7015","reported_seq":7168,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.666366+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:36.666366+0000","last_peered":"2024-04-26T02:29:36.666366+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:36.666366+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39170572,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":626777280,"num_objects":76518,"num_object_clones":0,"num_object_copies":306072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111039,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5332,"num_bytes_recovered":39264256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198379008,"data_stored":1198379008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111066,"ondisk_log_size":111066,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713192,"kb_used_data":308064,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93658648,"statfs":{"total":96636764160,"available":95906455552,"internally_reserved":0,"allocated":315457536,"data_stored":315263181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,5,6,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707856,"kb_used_data":307392,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93663984,"statfs":{"total":96636764160,"available":95911919616,"internally_reserved":0,"allocated":314769408,"data_stored":314575053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,0,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532408,"kb_used_data":252272,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93839432,"statfs":{"total":96636764160,"available":96091578368,"internally_reserved":0,"allocated":258326528,"data_stored":258132173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286846671},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706564,"kb_used_data":305660,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93665276,"statfs":{"total":96636764160,"available":95913242624,"internally_reserved":0,"allocated":312995840,"data_stored":312801485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,2,2,2,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312082432,"data_stored":312082432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257929216,"data_stored":257929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313892864,"data_stored":313892864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314474496,"data_stored":314474496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:40.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:41.872 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:41.873 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:41.963 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-04-26T02:29:40.570248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":621026496,"num_objects":75816,"num_object_clones":0,"num_object_copies":303264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111741,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5387,"num_bytes_recovered":39264256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111768,"ondisk_log_size":111768,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626184,"kb_used_data":1164516,"kb_used_omap":14,"kb_used_meta":1461617,"kb_avail":374861176,"statfs":{"total":386547056640,"available":383857844224,"internally_reserved":0,"allocated":1192464384,"data_stored":1191686964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1496696371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,10,7,25,13,14,2,0,1,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23535616,"num_objects":-2873,"num_object_clones":0,"num_object_copies":-11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007321"},"pg_stats":[{"pgid":"1.f","version":"36'6864","reported_seq":7007,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.875522+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:38.875522+0000","last_peered":"2024-04-26T02:29:38.875522+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:38.875522+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38174720,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6864,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7003","reported_seq":7144,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.588385+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:36.588385+0000","last_peered":"2024-04-26T02:29:36.588385+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:36.588385+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555596,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7001,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6922","reported_seq":7056,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:39.481262+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:39.481262+0000","last_peered":"2024-04-26T02:29:39.481262+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:39.481262+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6922,"log_dups_size":0,"ondisk_log_size":6922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38566174,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6921,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2710}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6821","reported_seq":8245,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.689905+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:36.689905+0000","last_peered":"2024-04-26T02:29:36.689905+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:36.689905+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38162956,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6926","reported_seq":7081,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.852057+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:38.852057+0000","last_peered":"2024-04-26T02:29:38.852057+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:38.852057+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6926,"log_dups_size":0,"ondisk_log_size":6926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38261260,"num_objects":4671,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6924,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'6958","reported_seq":7118,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.909448+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:38.909448+0000","last_peered":"2024-04-26T02:29:38.909448+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:38.909448+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38027264,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6958,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7016","reported_seq":11263,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:36.455489+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:36.455489+0000","last_peered":"2024-04-26T02:29:36.455489+0000","last_clean":"2024-04-26T02:29:36.455489+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:36.455489+0000","last_undegraded":"2024-04-26T02:29:36.455489+0000","last_fullsized":"2024-04-26T02:29:36.455489+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:41.031765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842892,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7143","reported_seq":7314,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.619538+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:36.619538+0000","last_peered":"2024-04-26T02:29:36.619538+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:36.619538+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39694860,"num_objects":4846,"num_object_clones":0,"num_object_copies":19384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7141,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'6982","reported_seq":7144,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.615078+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:38.615078+0000","last_peered":"2024-04-26T02:29:38.615078+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:38.615078+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38970392,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6897","reported_seq":7021,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:39.486764+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:39.486764+0000","last_peered":"2024-04-26T02:29:39.486764+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:39.486764+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38228492,"num_objects":4667,"num_object_clones":0,"num_object_copies":18668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'6988","reported_seq":7127,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.768042+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:38.768042+0000","last_peered":"2024-04-26T02:29:38.768042+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:38.768042+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38750994,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6899","reported_seq":7039,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.835533+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:38.835533+0000","last_peered":"2024-04-26T02:29:38.835533+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:38.835533+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38484230,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7059","reported_seq":7231,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:39.411918+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:39.411918+0000","last_peered":"2024-04-26T02:29:39.411918+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:39.411918+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7059,"log_dups_size":0,"ondisk_log_size":7059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38664454,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2655}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7188","reported_seq":7321,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:38.925108+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:38.925108+0000","last_peered":"2024-04-26T02:29:38.925108+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:38.925108+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40415756,"num_objects":4934,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7087","reported_seq":11292,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:38.055470+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:38.055470+0000","last_peered":"2024-04-26T02:29:38.055470+0000","last_clean":"2024-04-26T02:29:38.055470+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:38.055470+0000","last_undegraded":"2024-04-26T02:29:38.055470+0000","last_fullsized":"2024-04-26T02:29:38.055470+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:42.306447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39055884,"num_objects":4768,"num_object_clones":0,"num_object_copies":19072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7085,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7015","reported_seq":7168,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:36.666366+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:36.666366+0000","last_peered":"2024-04-26T02:29:36.666366+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:36.666366+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39170572,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":621026496,"num_objects":75816,"num_object_clones":0,"num_object_copies":303264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111741,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5387,"num_bytes_recovered":39264256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189220352,"data_stored":1189220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111768,"ondisk_log_size":111768,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117061,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696280,"kb_used_data":303628,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93675560,"statfs":{"total":96636764160,"available":95923773440,"internally_reserved":0,"allocated":310915072,"data_stored":310720717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,20,6,7,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690932,"kb_used_data":302956,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93680908,"statfs":{"total":96636764160,"available":95929249792,"internally_reserved":0,"allocated":310226944,"data_stored":310032589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,5,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532408,"kb_used_data":252272,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93839432,"statfs":{"total":96636764160,"available":96091578368,"internally_reserved":0,"allocated":258326528,"data_stored":258132173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286846671},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706564,"kb_used_data":305660,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93665276,"statfs":{"total":96636764160,"available":95913242624,"internally_reserved":0,"allocated":312995840,"data_stored":312801485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410513103},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,2,2,2,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312082432,"data_stored":312082432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257929216,"data_stored":257929216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":309342208,"data_stored":309342208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309866496,"data_stored":309866496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:41.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:41.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6864", 'reported_seq': 7007, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.875522+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:38.875522+0000', 'last_peered': '2024-04-26T02:29:38.875522+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:38.875522+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6864, 'log_dups_size': 0, 'ondisk_log_size': 6864, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38174720, 'num_objects': 4660, 'num_object_clones': 0, 'num_object_copies': 18640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6864, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:41.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7003", 'reported_seq': 7144, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:36.588385+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:36.588385+0000', 'last_peered': '2024-04-26T02:29:36.588385+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:36.588385+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7003, 'log_dups_size': 0, 'ondisk_log_size': 7003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39555596, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 19316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7001, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'6922", 'reported_seq': 7056, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:39.481262+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:39.481262+0000', 'last_peered': '2024-04-26T02:29:39.481262+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:39.481262+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6922, 'log_dups_size': 0, 'ondisk_log_size': 6922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38566174, 'num_objects': 4709, 'num_object_clones': 0, 'num_object_copies': 18836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4709, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6921, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2710}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6821", 'reported_seq': 8245, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:36.689905+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:36.689905+0000', 'last_peered': '2024-04-26T02:29:36.689905+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:36.689905+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6821, 'log_dups_size': 0, 'ondisk_log_size': 6821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38162956, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6819, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 680, 'num_bytes_recovered': 5390336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'6926", 'reported_seq': 7081, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.852057+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:38.852057+0000', 'last_peered': '2024-04-26T02:29:38.852057+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:38.852057+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6926, 'log_dups_size': 0, 'ondisk_log_size': 6926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38261260, 'num_objects': 4671, 'num_object_clones': 0, 'num_object_copies': 18684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6924, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2707}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'6958", 'reported_seq': 7118, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.909448+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:38.909448+0000', 'last_peered': '2024-04-26T02:29:38.909448+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:38.909448+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6958, 'log_dups_size': 0, 'ondisk_log_size': 6958, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38027264, 'num_objects': 4642, 'num_object_clones': 0, 'num_object_copies': 18568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1995, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6958, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7143", 'reported_seq': 7314, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:36.619538+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:36.619538+0000', 'last_peered': '2024-04-26T02:29:36.619538+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:36.619538+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7143, 'log_dups_size': 0, 'ondisk_log_size': 7143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39694860, 'num_objects': 4846, 'num_object_clones': 0, 'num_object_copies': 19384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7141, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2755}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:41.965 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'6982", 'reported_seq': 7144, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.615078+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:38.615078+0000', 'last_peered': '2024-04-26T02:29:38.615078+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:38.615078+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6982, 'log_dups_size': 0, 'ondisk_log_size': 6982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38970392, 'num_objects': 4758, 'num_object_clones': 0, 'num_object_copies': 19032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6979, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2804}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'6897", 'reported_seq': 7021, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:39.486764+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:39.486764+0000', 'last_peered': '2024-04-26T02:29:39.486764+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:39.486764+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38228492, 'num_objects': 4667, 'num_object_clones': 0, 'num_object_copies': 18668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2019, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6895, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2650}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'6988", 'reported_seq': 7127, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.768042+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:38.768042+0000', 'last_peered': '2024-04-26T02:29:38.768042+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:38.768042+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6988, 'log_dups_size': 0, 'ondisk_log_size': 6988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38750994, 'num_objects': 4731, 'num_object_clones': 0, 'num_object_copies': 18924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6985, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'6899", 'reported_seq': 7039, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.835533+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:38.835533+0000', 'last_peered': '2024-04-26T02:29:38.835533+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:38.835533+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6899, 'log_dups_size': 0, 'ondisk_log_size': 6899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38484230, 'num_objects': 4698, 'num_object_clones': 0, 'num_object_copies': 18792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1941, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6898, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7059", 'reported_seq': 7231, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:39.411918+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:39.411918+0000', 'last_peered': '2024-04-26T02:29:39.411918+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:39.411918+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7059, 'log_dups_size': 0, 'ondisk_log_size': 7059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38664454, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7058, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:41.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7188", 'reported_seq': 7321, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:38.925108+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:38.925108+0000', 'last_peered': '2024-04-26T02:29:38.925108+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:38.925108+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7188, 'log_dups_size': 0, 'ondisk_log_size': 7188, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40415756, 'num_objects': 4934, 'num_object_clones': 0, 'num_object_copies': 19736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7186, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:41.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:41.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7015", 'reported_seq': 7168, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:36.666366+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:36.666366+0000', 'last_peered': '2024-04-26T02:29:36.666366+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:36.666366+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7015, 'log_dups_size': 0, 'ondisk_log_size': 7015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39170572, 'num_objects': 4782, 'num_object_clones': 0, 'num_object_copies': 19128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7013, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:41.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:43.449 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:43.449 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27287/301772 objects degraded (9.042%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75443,"data_bytes":617970880,"bytes_used":2671824896,"bytes_avail":383875231744,"bytes_total":386547056640,"degraded_objects":27287,"degraded_total":301772,"degraded_ratio":0.090422570682502007,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":195782,"recovering_keys_per_sec":0,"num_objects_recovered":394,"num_bytes_recovered":2351104,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:43.517 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:43.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:44.739 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:44.739 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:44.838 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-04-26T02:29:44.576252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":611925184,"num_objects":74705,"num_object_clones":0,"num_object_copies":298820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112852,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":39952384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112879,"ondisk_log_size":112879,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2616608,"kb_used_data":1150780,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374870752,"statfs":{"total":386547056640,"available":383867650048,"internally_reserved":0,"allocated":1178398720,"data_stored":1177621300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1500956211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,5,5,7,9,10,0,4,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23994368,"num_objects":-2929,"num_object_clones":0,"num_object_copies":-11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2929,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010923"},"pg_stats":[{"pgid":"1.f","version":"36'6920","reported_seq":7065,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.980466+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:43.980466+0000","last_peered":"2024-04-26T02:29:43.980466+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:43.980466+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37715968,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2636}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7081","reported_seq":7226,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:41.746671+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:41.746671+0000","last_peered":"2024-04-26T02:29:41.746671+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:41.746671+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7081,"log_dups_size":0,"ondisk_log_size":7081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38916620,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6986","reported_seq":7126,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.435754+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:44.435754+0000","last_peered":"2024-04-26T02:29:44.435754+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:44.435754+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38041886,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6985,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6895","reported_seq":8486,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:41.703092+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:41.703092+0000","last_peered":"2024-04-26T02:29:41.703092+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:41.703092+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37556748,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6893,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":6078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3377}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6994","reported_seq":7149,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.834050+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:43.834050+0000","last_peered":"2024-04-26T02:29:43.834050+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:43.834050+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37704204,"num_objects":4603,"num_object_clones":0,"num_object_copies":18412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7018","reported_seq":7183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.043273+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:44.043273+0000","last_peered":"2024-04-26T02:29:44.043273+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:44.043273+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37535744,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7083","reported_seq":11330,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:41.483228+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:41.483228+0000","last_peered":"2024-04-26T02:29:41.483228+0000","last_clean":"2024-04-26T02:29:41.483228+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:41.483228+0000","last_undegraded":"2024-04-26T02:29:41.483228+0000","last_fullsized":"2024-04-26T02:29:41.483228+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38294028,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7215","reported_seq":7398,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:41.708541+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:41.708541+0000","last_peered":"2024-04-26T02:29:41.708541+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:41.708541+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7215,"log_dups_size":0,"ondisk_log_size":7215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39105036,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7043","reported_seq":7207,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.850958+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:43.850958+0000","last_peered":"2024-04-26T02:29:43.850958+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:43.850958+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38470680,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6967","reported_seq":7093,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.428816+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:44.428816+0000","last_peered":"2024-04-26T02:29:44.428816+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:44.428816+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6967,"log_dups_size":0,"ondisk_log_size":6967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37655052,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6965,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7048","reported_seq":7193,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.953253+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:43.953253+0000","last_peered":"2024-04-26T02:29:43.953253+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:43.953253+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38259474,"num_objects":4671,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6984","reported_seq":7128,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.981785+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:43.981785+0000","last_peered":"2024-04-26T02:29:43.981785+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:43.981785+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6984,"log_dups_size":0,"ondisk_log_size":6984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37787910,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6983,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7119","reported_seq":7293,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.465915+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:44.465915+0000","last_peered":"2024-04-26T02:29:44.465915+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:44.465915+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7119,"log_dups_size":0,"ondisk_log_size":7119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38172934,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7269","reported_seq":7405,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.007433+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:44.007433+0000","last_peered":"2024-04-26T02:29:44.007433+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:44.007433+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39752204,"num_objects":4853,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7160","reported_seq":11365,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:43.057326+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:43.057326+0000","last_peered":"2024-04-26T02:29:43.057326+0000","last_clean":"2024-04-26T02:29:43.057326+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:43.057326+0000","last_undegraded":"2024-04-26T02:29:43.057326+0000","last_fullsized":"2024-04-26T02:29:43.057326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:47.346989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38457868,"num_objects":4695,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7097","reported_seq":7257,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:41.710095+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:41.710095+0000","last_peered":"2024-04-26T02:29:41.710095+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:41.710095+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7097,"log_dups_size":0,"ondisk_log_size":7097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38498828,"num_objects":4700,"num_object_clones":0,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":611925184,"num_objects":74705,"num_object_clones":0,"num_object_copies":298820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112852,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5497,"num_bytes_recovered":39952384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1175031808,"data_stored":1175031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112879,"ondisk_log_size":112879,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695980,"kb_used_data":299172,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93675860,"statfs":{"total":96636764160,"available":95924080640,"internally_reserved":0,"allocated":306352128,"data_stored":306157773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,3,3,2,3,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690804,"kb_used_data":298664,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93681036,"statfs":{"total":96636764160,"available":95929380864,"internally_reserved":0,"allocated":305831936,"data_stored":305637581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,5,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540240,"kb_used_data":251784,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93831600,"statfs":{"total":96636764160,"available":96083558400,"internally_reserved":0,"allocated":257826816,"data_stored":257632461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689584,"kb_used_data":301160,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93682256,"statfs":{"total":96636764160,"available":95930630144,"internally_reserved":0,"allocated":308387840,"data_stored":308193485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397733583},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":307326976,"data_stored":307326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257458176,"data_stored":257458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304922624,"data_stored":304922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305324032,"data_stored":305324032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:47.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:49.441 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:49.442 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:49.587 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-26T02:29:48.577219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608533696,"num_objects":74291,"num_object_clones":0,"num_object_copies":297164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113266,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5578,"num_bytes_recovered":40566784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113293,"ondisk_log_size":113293,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2624212,"kb_used_data":1145896,"kb_used_omap":14,"kb_used_meta":1478257,"kb_avail":374863148,"statfs":{"total":386547056640,"available":383859863552,"internally_reserved":0,"allocated":1173397504,"data_stored":1172620084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1513735731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,11,11,8,8,1,5,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21766144,"num_objects":-2657,"num_object_clones":0,"num_object_copies":-10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008601"},"pg_stats":[{"pgid":"1.f","version":"36'6920","reported_seq":7065,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.980466+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:43.980466+0000","last_peered":"2024-04-26T02:29:43.980466+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:43.980466+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37715968,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2636}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7156","reported_seq":7307,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.849208+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:46.849208+0000","last_peered":"2024-04-26T02:29:46.849208+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:46.849208+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302220,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'6986","reported_seq":7126,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.435754+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:44.435754+0000","last_peered":"2024-04-26T02:29:44.435754+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:44.435754+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38041886,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6985,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6964","reported_seq":8707,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.764137+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:46.764137+0000","last_peered":"2024-04-26T02:29:46.764137+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:46.764137+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36991500,"num_objects":4516,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":839,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'6994","reported_seq":7149,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.834050+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:43.834050+0000","last_peered":"2024-04-26T02:29:43.834050+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:43.834050+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37704204,"num_objects":4603,"num_object_clones":0,"num_object_copies":18412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7018","reported_seq":7183,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.043273+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:44.043273+0000","last_peered":"2024-04-26T02:29:44.043273+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:44.043273+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37535744,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7148","reported_seq":11395,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:46.805648+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:46.805648+0000","last_peered":"2024-04-26T02:29:46.805648+0000","last_clean":"2024-04-26T02:29:46.805648+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:46.805648+0000","last_undegraded":"2024-04-26T02:29:46.805648+0000","last_fullsized":"2024-04-26T02:29:46.805648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:46.966553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37761548,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7286","reported_seq":7473,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.838656+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:46.838656+0000","last_peered":"2024-04-26T02:29:46.838656+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:46.838656+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38523404,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7043","reported_seq":7207,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.850958+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:43.850958+0000","last_peered":"2024-04-26T02:29:43.850958+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:43.850958+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38470680,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2744}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'6967","reported_seq":7093,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.428816+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:44.428816+0000","last_peered":"2024-04-26T02:29:44.428816+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:44.428816+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6967,"log_dups_size":0,"ondisk_log_size":6967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37655052,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6965,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7048","reported_seq":7193,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.953253+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:43.953253+0000","last_peered":"2024-04-26T02:29:43.953253+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:43.953253+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38259474,"num_objects":4671,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'6984","reported_seq":7128,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:43.981785+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:43.981785+0000","last_peered":"2024-04-26T02:29:43.981785+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:43.981785+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6984,"log_dups_size":0,"ondisk_log_size":6984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37787910,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6983,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7119","reported_seq":7293,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.465915+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:44.465915+0000","last_peered":"2024-04-26T02:29:44.465915+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:44.465915+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7119,"log_dups_size":0,"ondisk_log_size":7119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38172934,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7269","reported_seq":7405,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:44.007433+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:44.007433+0000","last_peered":"2024-04-26T02:29:44.007433+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:44.007433+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39752204,"num_objects":4853,"num_object_clones":0,"num_object_copies":19412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7231","reported_seq":11436,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:48.086789+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:48.086789+0000","last_peered":"2024-04-26T02:29:48.086789+0000","last_clean":"2024-04-26T02:29:48.086789+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:48.086789+0000","last_undegraded":"2024-04-26T02:29:48.086789+0000","last_fullsized":"2024-04-26T02:29:48.086789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:52.458958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876236,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7160","reported_seq":7322,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.687897+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:46.687897+0000","last_peered":"2024-04-26T02:29:46.687897+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:46.687897+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37982732,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":608533696,"num_objects":74291,"num_object_clones":0,"num_object_copies":297164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113266,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5578,"num_bytes_recovered":40566784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1169842176,"data_stored":1169842176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113293,"ondisk_log_size":113293,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695980,"kb_used_data":299172,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93675860,"statfs":{"total":96636764160,"available":95924080640,"internally_reserved":0,"allocated":306352128,"data_stored":306157773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,3,3,2,3,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690804,"kb_used_data":298664,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93681036,"statfs":{"total":96636764160,"available":95929380864,"internally_reserved":0,"allocated":305831936,"data_stored":305637581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,5,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543872,"kb_used_data":251256,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93827968,"statfs":{"total":96636764160,"available":96079839232,"internally_reserved":0,"allocated":257286144,"data_stored":257091789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693556,"kb_used_data":296804,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93678284,"statfs":{"total":96636764160,"available":95926562816,"internally_reserved":0,"allocated":303927296,"data_stored":303732941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,2,0,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302682112,"data_stored":302682112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256913408,"data_stored":256913408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304922624,"data_stored":304922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305324032,"data_stored":305324032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:49.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:51.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:51.112 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:51.156 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-04-26T02:29:50.577714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602586304,"num_objects":73565,"num_object_clones":0,"num_object_copies":294260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113992,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5609,"num_bytes_recovered":40566784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114019,"ondisk_log_size":114019,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651084,"kb_used_data":1136676,"kb_used_omap":14,"kb_used_meta":1514353,"kb_avail":374836276,"statfs":{"total":386547056640,"available":383832346624,"internally_reserved":0,"allocated":1163956224,"data_stored":1163178804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1550698035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,11,13,5,8,2,1,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24190976,"num_objects":-2953,"num_object_clones":0,"num_object_copies":-11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008624"},"pg_stats":[{"pgid":"1.f","version":"36'6979","reported_seq":7130,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.901943+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:48.901943+0000","last_peered":"2024-04-26T02:29:48.901943+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:48.901943+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37232640,"num_objects":4545,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7156","reported_seq":7307,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.849208+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:46.849208+0000","last_peered":"2024-04-26T02:29:46.849208+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:46.849208+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302220,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7054","reported_seq":7195,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.505414+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:49.505414+0000","last_peered":"2024-04-26T02:29:49.505414+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:49.505414+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37484830,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7053,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'6964","reported_seq":8707,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.764137+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:46.764137+0000","last_peered":"2024-04-26T02:29:46.764137+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:46.764137+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6964,"log_dups_size":0,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36991500,"num_objects":4516,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":839,"num_bytes_recovered":6692864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7064","reported_seq":7225,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.987214+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:48.987214+0000","last_peered":"2024-04-26T02:29:48.987214+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:48.987214+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37130764,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7094","reported_seq":7265,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.004895+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:49.004895+0000","last_peered":"2024-04-26T02:29:49.004895+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:49.004895+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7094,"log_dups_size":0,"ondisk_log_size":7094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36913152,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7094,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7148","reported_seq":11395,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:46.805648+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:46.805648+0000","last_peered":"2024-04-26T02:29:46.805648+0000","last_clean":"2024-04-26T02:29:46.805648+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:46.805648+0000","last_undegraded":"2024-04-26T02:29:46.805648+0000","last_fullsized":"2024-04-26T02:29:46.805648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:46.966553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37761548,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7286","reported_seq":7473,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.838656+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:46.838656+0000","last_peered":"2024-04-26T02:29:46.838656+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:46.838656+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38523404,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7120","reported_seq":7288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.115628+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:49.115628+0000","last_peered":"2024-04-26T02:29:49.115628+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:49.115628+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37839896,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7047","reported_seq":7175,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.392849+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:49.392849+0000","last_peered":"2024-04-26T02:29:49.392849+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:49.392849+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36999692,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7120","reported_seq":7273,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.923242+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:48.923242+0000","last_peered":"2024-04-26T02:29:48.923242+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:48.923242+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37669650,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7054","reported_seq":7204,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.909729+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:48.909729+0000","last_peered":"2024-04-26T02:29:48.909729+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:48.909729+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214470,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7197","reported_seq":7377,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.502629+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:49.502629+0000","last_peered":"2024-04-26T02:29:49.502629+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:49.502629+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37533958,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7345","reported_seq":7497,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.004335+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:49.004335+0000","last_peered":"2024-04-26T02:29:49.004335+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:49.004335+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129612,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2753}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7231","reported_seq":11436,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:48.086789+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:48.086789+0000","last_peered":"2024-04-26T02:29:48.086789+0000","last_clean":"2024-04-26T02:29:48.086789+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:48.086789+0000","last_undegraded":"2024-04-26T02:29:48.086789+0000","last_fullsized":"2024-04-26T02:29:48.086789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:52.458958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876236,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7160","reported_seq":7322,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:46.687897+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:46.687897+0000","last_peered":"2024-04-26T02:29:46.687897+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:46.687897+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37982732,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":602586304,"num_objects":73565,"num_object_clones":0,"num_object_copies":294260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113992,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5609,"num_bytes_recovered":40566784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1160216576,"data_stored":1160216576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114019,"ondisk_log_size":114019,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701092,"kb_used_data":294556,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93670748,"statfs":{"total":96636764160,"available":95918845952,"internally_reserved":0,"allocated":301625344,"data_stored":301430989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,1,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712564,"kb_used_data":294060,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93659276,"statfs":{"total":96636764160,"available":95907098624,"internally_reserved":0,"allocated":301117440,"data_stored":300923085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,0,5,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543872,"kb_used_data":251256,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93827968,"statfs":{"total":96636764160,"available":96079839232,"internally_reserved":0,"allocated":257286144,"data_stored":257091789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693556,"kb_used_data":296804,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93678284,"statfs":{"total":96636764160,"available":95926562816,"internally_reserved":0,"allocated":303927296,"data_stored":303732941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406253263},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,2,0,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302682112,"data_stored":302682112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256913408,"data_stored":256913408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":300105728,"data_stored":300105728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300515328,"data_stored":300515328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'6979", 'reported_seq': 7130, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:48.901943+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:48.901943+0000', 'last_peered': '2024-04-26T02:29:48.901943+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:48.901943+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6979, 'log_dups_size': 0, 'ondisk_log_size': 6979, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37232640, 'num_objects': 4545, 'num_object_clones': 0, 'num_object_copies': 18180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6979, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2580}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7156", 'reported_seq': 7307, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:46.849208+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:46.849208+0000', 'last_peered': '2024-04-26T02:29:46.849208+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:46.849208+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7156, 'log_dups_size': 0, 'ondisk_log_size': 7156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38302220, 'num_objects': 4676, 'num_object_clones': 0, 'num_object_copies': 18704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7154, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2640}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:51.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7054", 'reported_seq': 7195, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.505414+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:49.505414+0000', 'last_peered': '2024-04-26T02:29:49.505414+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:49.505414+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7054, 'log_dups_size': 0, 'ondisk_log_size': 7054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37484830, 'num_objects': 4577, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1996, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4577, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7053, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2582}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'6964", 'reported_seq': 8707, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:46.764137+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:46.764137+0000', 'last_peered': '2024-04-26T02:29:46.764137+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:46.764137+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 6964, 'log_dups_size': 0, 'ondisk_log_size': 6964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36991500, 'num_objects': 4516, 'num_object_clones': 0, 'num_object_copies': 18064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1133, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6962, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 839, 'num_bytes_recovered': 6692864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7064", 'reported_seq': 7225, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:48.987214+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:48.987214+0000', 'last_peered': '2024-04-26T02:29:48.987214+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:48.987214+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7064, 'log_dups_size': 0, 'ondisk_log_size': 7064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37130764, 'num_objects': 4533, 'num_object_clones': 0, 'num_object_copies': 18132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7062, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7094", 'reported_seq': 7265, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.004895+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:49.004895+0000', 'last_peered': '2024-04-26T02:29:49.004895+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:49.004895+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7094, 'log_dups_size': 0, 'ondisk_log_size': 7094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36913152, 'num_objects': 4506, 'num_object_clones': 0, 'num_object_copies': 18024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7094, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7286", 'reported_seq': 7473, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:46.838656+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:46.838656+0000', 'last_peered': '2024-04-26T02:29:46.838656+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:46.838656+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7286, 'log_dups_size': 0, 'ondisk_log_size': 7286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38523404, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7284, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:51.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7120", 'reported_seq': 7288, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.115628+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:49.115628+0000', 'last_peered': '2024-04-26T02:29:49.115628+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:49.115628+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7120, 'log_dups_size': 0, 'ondisk_log_size': 7120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37839896, 'num_objects': 4620, 'num_object_clones': 0, 'num_object_copies': 18480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1954, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7117, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2669}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7047", 'reported_seq': 7175, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.392849+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:49.392849+0000', 'last_peered': '2024-04-26T02:29:49.392849+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:49.392849+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7047, 'log_dups_size': 0, 'ondisk_log_size': 7047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36999692, 'num_objects': 4517, 'num_object_clones': 0, 'num_object_copies': 18068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7045, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2502}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7120", 'reported_seq': 7273, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:48.923242+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:48.923242+0000', 'last_peered': '2024-04-26T02:29:48.923242+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:48.923242+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7120, 'log_dups_size': 0, 'ondisk_log_size': 7120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37669650, 'num_objects': 4599, 'num_object_clones': 0, 'num_object_copies': 18396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7117, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7054", 'reported_seq': 7204, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:48.909729+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:48.909729+0000', 'last_peered': '2024-04-26T02:29:48.909729+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:48.909729+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7054, 'log_dups_size': 0, 'ondisk_log_size': 7054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37214470, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 18172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1936, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7053, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7197", 'reported_seq': 7377, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.502629+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:49.502629+0000', 'last_peered': '2024-04-26T02:29:49.502629+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:49.502629+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7197, 'log_dups_size': 0, 'ondisk_log_size': 7197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37533958, 'num_objects': 4582, 'num_object_clones': 0, 'num_object_copies': 18328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7196, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7345", 'reported_seq': 7497, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:49.004335+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:49.004335+0000', 'last_peered': '2024-04-26T02:29:49.004335+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:49.004335+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7345, 'log_dups_size': 0, 'ondisk_log_size': 7345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39129612, 'num_objects': 4777, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7343, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:51.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7160", 'reported_seq': 7322, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:46.687897+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:46.687897+0000', 'last_peered': '2024-04-26T02:29:46.687897+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:46.687897+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7160, 'log_dups_size': 0, 'ondisk_log_size': 7160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37982732, 'num_objects': 4637, 'num_object_clones': 0, 'num_object_copies': 18548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7158, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2619}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:51.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:29:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:52.657 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27160/294260 objects degraded (9.230%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":389,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73565,"data_bytes":602586304,"bytes_used":2714710016,"bytes_avail":383832346624,"bytes_total":386547056640,"degraded_objects":27160,"degraded_total":294260,"degraded_ratio":0.092299327125671168,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":108466,"recovering_keys_per_sec":0,"num_objects_recovered":277,"num_bytes_recovered":1302528,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:29:52.772 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:29:52.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:53.585 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:53.660 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-04-26T02:29:52.578232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599456960,"num_objects":73183,"num_object_clones":0,"num_object_copies":292732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114374,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5698,"num_bytes_recovered":41238528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114401,"ondisk_log_size":114401,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2668572,"kb_used_data":1131964,"kb_used_omap":14,"kb_used_meta":1536561,"kb_avail":374818788,"statfs":{"total":386547056640,"available":383814438912,"internally_reserved":0,"allocated":1159131136,"data_stored":1158353716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1573439027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,7,11,10,5,9,1,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21569536,"num_objects":-2633,"num_object_clones":0,"num_object_copies":-10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007984"},"pg_stats":[{"pgid":"1.f","version":"36'6979","reported_seq":7130,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.901943+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:48.901943+0000","last_peered":"2024-04-26T02:29:48.901943+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:48.901943+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37232640,"num_objects":4545,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7238","reported_seq":7395,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.748895+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:51.748895+0000","last_peered":"2024-04-26T02:29:51.748895+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:51.748895+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37630476,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7054","reported_seq":7195,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.505414+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:49.505414+0000","last_peered":"2024-04-26T02:29:49.505414+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:49.505414+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37484830,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7053,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7037","reported_seq":8944,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.906975+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:51.906975+0000","last_peered":"2024-04-26T02:29:51.906975+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:51.906975+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7037,"log_dups_size":0,"ondisk_log_size":7037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36393484,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7035,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7064","reported_seq":7225,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.987214+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:48.987214+0000","last_peered":"2024-04-26T02:29:48.987214+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:48.987214+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37130764,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7094","reported_seq":7265,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.004895+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:49.004895+0000","last_peered":"2024-04-26T02:29:49.004895+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:49.004895+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7094,"log_dups_size":0,"ondisk_log_size":7094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36913152,"num_objects":4506,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7094,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7211","reported_seq":11458,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:51.141157+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:51.141157+0000","last_peered":"2024-04-26T02:29:51.141157+0000","last_clean":"2024-04-26T02:29:51.141157+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:51.141157+0000","last_undegraded":"2024-04-26T02:29:51.141157+0000","last_fullsized":"2024-04-26T02:29:51.141157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:52.892094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37245452,"num_objects":4547,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7371","reported_seq":7562,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.736038+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:51.736038+0000","last_peered":"2024-04-26T02:29:51.736038+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:51.736038+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37827084,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7120","reported_seq":7288,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.115628+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:49.115628+0000","last_peered":"2024-04-26T02:29:49.115628+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:49.115628+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37839896,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7047","reported_seq":7175,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.392849+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:49.392849+0000","last_peered":"2024-04-26T02:29:49.392849+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:49.392849+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36999692,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7120","reported_seq":7273,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.923242+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:48.923242+0000","last_peered":"2024-04-26T02:29:48.923242+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:48.923242+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37669650,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7054","reported_seq":7204,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:48.909729+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:48.909729+0000","last_peered":"2024-04-26T02:29:48.909729+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:48.909729+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37214470,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7197","reported_seq":7377,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.502629+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:49.502629+0000","last_peered":"2024-04-26T02:29:49.502629+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:49.502629+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37533958,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7345","reported_seq":7497,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:49.004335+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:49.004335+0000","last_peered":"2024-04-26T02:29:49.004335+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:49.004335+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129612,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2753}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7231","reported_seq":11436,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:48.086789+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:48.086789+0000","last_peered":"2024-04-26T02:29:48.086789+0000","last_clean":"2024-04-26T02:29:48.086789+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:48.086789+0000","last_undegraded":"2024-04-26T02:29:48.086789+0000","last_fullsized":"2024-04-26T02:29:48.086789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:52.458958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876236,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7239","reported_seq":7403,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.690105+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:51.690105+0000","last_peered":"2024-04-26T02:29:51.690105+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:51.690105+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7239,"log_dups_size":0,"ondisk_log_size":7239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335564,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":599456960,"num_objects":73183,"num_object_clones":0,"num_object_copies":292732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114374,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5698,"num_bytes_recovered":41238528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155493888,"data_stored":1155493888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114401,"ondisk_log_size":114401,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701092,"kb_used_data":294556,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93670748,"statfs":{"total":96636764160,"available":95918845952,"internally_reserved":0,"allocated":301625344,"data_stored":301430989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":416280272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,1,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712564,"kb_used_data":294060,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93659276,"statfs":{"total":96636764160,"available":95907098624,"internally_reserved":0,"allocated":301117440,"data_stored":300923085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,0,5,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543872,"kb_used_data":251256,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93827968,"statfs":{"total":96636764160,"available":96079839232,"internally_reserved":0,"allocated":257286144,"data_stored":257091789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711044,"kb_used_data":292092,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93660796,"statfs":{"total":96636764160,"available":95908655104,"internally_reserved":0,"allocated":299102208,"data_stored":298907853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297959424,"data_stored":297959424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256913408,"data_stored":256913408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":300105728,"data_stored":300105728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300515328,"data_stored":300515328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:56.662 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:57.757 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:57.761 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:57.846 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-04-26T02:29:56.579254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593206464,"num_objects":72420,"num_object_clones":0,"num_object_copies":289680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115137,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5702,"num_bytes_recovered":41238528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115164,"ondisk_log_size":115164,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660728,"kb_used_data":1122708,"kb_used_omap":14,"kb_used_meta":1537969,"kb_avail":374826632,"statfs":{"total":386547056640,"available":383822471168,"internally_reserved":0,"allocated":1149652992,"data_stored":1148875572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1574880819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,6,15,3,7,5,1,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18718720,"num_objects":-2285,"num_object_clones":0,"num_object_copies":-9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2285,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":1286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"1.f","version":"36'7045","reported_seq":7196,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:53.979725+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:53.979725+0000","last_peered":"2024-04-26T02:29:53.979725+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:53.979725+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7238","reported_seq":7395,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.748895+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:51.748895+0000","last_peered":"2024-04-26T02:29:51.748895+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:51.748895+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37630476,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7121","reported_seq":7263,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.472658+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:54.472658+0000","last_peered":"2024-04-26T02:29:54.472658+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:54.472658+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36935966,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7120,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7037","reported_seq":8944,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.906975+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:51.906975+0000","last_peered":"2024-04-26T02:29:51.906975+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:51.906975+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7037,"log_dups_size":0,"ondisk_log_size":7037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36393484,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7035,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7135","reported_seq":7296,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.014135+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:54.014135+0000","last_peered":"2024-04-26T02:29:54.014135+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:54.014135+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36549132,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7158","reported_seq":7329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.083177+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:54.083177+0000","last_peered":"2024-04-26T02:29:54.083177+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:54.083177+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7158,"log_dups_size":0,"ondisk_log_size":7158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7211","reported_seq":11458,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:51.141157+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:51.141157+0000","last_peered":"2024-04-26T02:29:51.141157+0000","last_clean":"2024-04-26T02:29:51.141157+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:51.141157+0000","last_undegraded":"2024-04-26T02:29:51.141157+0000","last_fullsized":"2024-04-26T02:29:51.141157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:52.892094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37245452,"num_objects":4547,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7371","reported_seq":7562,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.736038+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:51.736038+0000","last_peered":"2024-04-26T02:29:51.736038+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:51.736038+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37827084,"num_objects":4618,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7201","reported_seq":7372,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.094434+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:54.094434+0000","last_peered":"2024-04-26T02:29:54.094434+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:54.094434+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7201,"log_dups_size":0,"ondisk_log_size":7201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37176344,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7120","reported_seq":7246,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.510708+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:54.510708+0000","last_peered":"2024-04-26T02:29:54.510708+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:54.510708+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36401676,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7183","reported_seq":7338,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.011434+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:54.011434+0000","last_peered":"2024-04-26T02:29:54.011434+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:54.011434+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37153554,"num_objects":4536,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7180,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7116","reported_seq":7266,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.098117+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:54.098117+0000","last_peered":"2024-04-26T02:29:54.098117+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:54.098117+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36706566,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7115,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7266","reported_seq":7446,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.579416+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:54.579416+0000","last_peered":"2024-04-26T02:29:54.579416+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:54.579416+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36968710,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7426","reported_seq":7579,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:53.992626+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:53.992626+0000","last_peered":"2024-04-26T02:29:53.992626+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:53.992626+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38466060,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7297","reported_seq":11502,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:53.021988+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:53.021988+0000","last_peered":"2024-04-26T02:29:53.021988+0000","last_clean":"2024-04-26T02:29:53.021988+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:53.021988+0000","last_undegraded":"2024-04-26T02:29:53.021988+0000","last_fullsized":"2024-04-26T02:29:53.021988+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:57.546563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335564,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7295,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7239","reported_seq":7403,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:51.690105+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:51.690105+0000","last_peered":"2024-04-26T02:29:51.690105+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:51.690105+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7239,"log_dups_size":0,"ondisk_log_size":7239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335564,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":593206464,"num_objects":72420,"num_object_clones":0,"num_object_copies":289680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115137,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5702,"num_bytes_recovered":41238528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1146036224,"data_stored":1146036224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115164,"ondisk_log_size":115164,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700792,"kb_used_data":290092,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93671048,"statfs":{"total":96636764160,"available":95919153152,"internally_reserved":0,"allocated":297054208,"data_stored":296859853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":420540112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,1,3,3,1,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695716,"kb_used_data":289692,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93676124,"statfs":{"total":96636764160,"available":95924350976,"internally_reserved":0,"allocated":296644608,"data_stored":296450253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553176,"kb_used_data":250832,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93818664,"statfs":{"total":96636764160,"available":96070311936,"internally_reserved":0,"allocated":256851968,"data_stored":256657613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711044,"kb_used_data":292092,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93660796,"statfs":{"total":96636764160,"available":95908655104,"internally_reserved":0,"allocated":299102208,"data_stored":298907853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297959424,"data_stored":297959424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256479232,"data_stored":256479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295579648,"data_stored":295579648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296017920,"data_stored":296017920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:57.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:29:59.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:29:59.150 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:29:59.276 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-04-26T02:29:58.579767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590003392,"num_objects":72029,"num_object_clones":0,"num_object_copies":288116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115528,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5830,"num_bytes_recovered":42237952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115555,"ondisk_log_size":115555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647800,"kb_used_data":1118088,"kb_used_omap":14,"kb_used_meta":1529649,"kb_avail":374839560,"statfs":{"total":386547056640,"available":383835709440,"internally_reserved":0,"allocated":1144922112,"data_stored":1144144692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1566361139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,3,13,4,9,4,1,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21921792,"num_objects":-2676,"num_object_clones":0,"num_object_copies":-10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":333,"num_bytes_recovered":2285568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003023"},"pg_stats":[{"pgid":"1.f","version":"36'7045","reported_seq":7196,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:53.979725+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:53.979725+0000","last_peered":"2024-04-26T02:29:53.979725+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:53.979725+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7298","reported_seq":7457,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.875477+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:56.875477+0000","last_peered":"2024-04-26T02:29:56.875477+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:56.875477+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7298,"log_dups_size":0,"ondisk_log_size":7298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37138956,"num_objects":4534,"num_object_clones":0,"num_object_copies":18136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7121","reported_seq":7263,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.472658+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:54.472658+0000","last_peered":"2024-04-26T02:29:54.472658+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:54.472658+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36935966,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7120,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7092","reported_seq":9243,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.938862+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:56.938862+0000","last_peered":"2024-04-26T02:29:56.938862+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:56.938862+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7092,"log_dups_size":0,"ondisk_log_size":7092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35942924,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":8364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7135","reported_seq":7296,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.014135+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:54.014135+0000","last_peered":"2024-04-26T02:29:54.014135+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:54.014135+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7135,"log_dups_size":0,"ondisk_log_size":7135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36549132,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2501}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7158","reported_seq":7329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.083177+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:54.083177+0000","last_peered":"2024-04-26T02:29:54.083177+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:54.083177+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7158,"log_dups_size":0,"ondisk_log_size":7158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7279","reported_seq":11526,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:56.826388+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:56.826388+0000","last_peered":"2024-04-26T02:29:56.826388+0000","last_clean":"2024-04-26T02:29:56.826388+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:56.826388+0000","last_undegraded":"2024-04-26T02:29:56.826388+0000","last_fullsized":"2024-04-26T02:29:56.826388+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:57.947922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36688396,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7277,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7441","reported_seq":7642,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.674521+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:56.674521+0000","last_peered":"2024-04-26T02:29:56.674521+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:56.674521+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253644,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7201","reported_seq":7372,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.094434+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:54.094434+0000","last_peered":"2024-04-26T02:29:54.094434+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:54.094434+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7201,"log_dups_size":0,"ondisk_log_size":7201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37176344,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7120","reported_seq":7246,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.510708+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:54.510708+0000","last_peered":"2024-04-26T02:29:54.510708+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:54.510708+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36401676,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7183","reported_seq":7338,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.011434+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:54.011434+0000","last_peered":"2024-04-26T02:29:54.011434+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:54.011434+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37153554,"num_objects":4536,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7180,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7116","reported_seq":7266,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.098117+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:54.098117+0000","last_peered":"2024-04-26T02:29:54.098117+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:54.098117+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36706566,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7115,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7266","reported_seq":7446,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:54.579416+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:54.579416+0000","last_peered":"2024-04-26T02:29:54.579416+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:54.579416+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36968710,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7426","reported_seq":7579,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:53.992626+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:53.992626+0000","last_peered":"2024-04-26T02:29:53.992626+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:53.992626+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38466060,"num_objects":4696,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7359","reported_seq":11563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:58.137713+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:58.137713+0000","last_peered":"2024-04-26T02:29:58.137713+0000","last_clean":"2024-04-26T02:29:58.137713+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:58.137713+0000","last_undegraded":"2024-04-26T02:29:58.137713+0000","last_fullsized":"2024-04-26T02:29:58.137713+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36827660,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7315","reported_seq":7481,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.898824+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:56.898824+0000","last_peered":"2024-04-26T02:29:56.898824+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:56.898824+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36712972,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590003392,"num_objects":72029,"num_object_clones":0,"num_object_copies":288116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115528,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5830,"num_bytes_recovered":42237952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141325824,"data_stored":1141325824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115555,"ondisk_log_size":115555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117064,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700792,"kb_used_data":290092,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93671048,"statfs":{"total":96636764160,"available":95919153152,"internally_reserved":0,"allocated":297054208,"data_stored":296859853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":420540112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,1,3,3,1,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695716,"kb_used_data":289692,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93676124,"statfs":{"total":96636764160,"available":95924350976,"internally_reserved":0,"allocated":296644608,"data_stored":296450253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557068,"kb_used_data":250560,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93814772,"statfs":{"total":96636764160,"available":96066326528,"internally_reserved":0,"allocated":256573440,"data_stored":256379085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694224,"kb_used_data":287744,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93677616,"statfs":{"total":96636764160,"available":95925878784,"internally_reserved":0,"allocated":294649856,"data_stored":294455501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293478400,"data_stored":293478400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256249856,"data_stored":256249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295579648,"data_stored":295579648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":296017920,"data_stored":296017920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7045", 'reported_seq': 7196, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:53.979725+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:29:53.979725+0000', 'last_peered': '2024-04-26T02:29:53.979725+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:29:53.979725+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7045, 'log_dups_size': 0, 'ondisk_log_size': 7045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36691968, 'num_objects': 4479, 'num_object_clones': 0, 'num_object_copies': 17916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7045, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7298", 'reported_seq': 7457, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:56.875477+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:29:56.875477+0000', 'last_peered': '2024-04-26T02:29:56.875477+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:29:56.875477+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7298, 'log_dups_size': 0, 'ondisk_log_size': 7298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37138956, 'num_objects': 4534, 'num_object_clones': 0, 'num_object_copies': 18136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7296, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7121", 'reported_seq': 7263, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.472658+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:29:54.472658+0000', 'last_peered': '2024-04-26T02:29:54.472658+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:29:54.472658+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7121, 'log_dups_size': 0, 'ondisk_log_size': 7121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36935966, 'num_objects': 4510, 'num_object_clones': 0, 'num_object_copies': 18040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1996, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4510, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7120, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2515}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:29:59.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'7092", 'reported_seq': 9243, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:56.938862+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:29:56.938862+0000', 'last_peered': '2024-04-26T02:29:56.938862+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:29:56.938862+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7092, 'log_dups_size': 0, 'ondisk_log_size': 7092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35942924, 'num_objects': 4388, 'num_object_clones': 0, 'num_object_copies': 17552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7090, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1043, 'num_bytes_recovered': 8364032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3459}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7135", 'reported_seq': 7296, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.014135+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:29:54.014135+0000', 'last_peered': '2024-04-26T02:29:54.014135+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:29:54.014135+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7135, 'log_dups_size': 0, 'ondisk_log_size': 7135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36549132, 'num_objects': 4462, 'num_object_clones': 0, 'num_object_copies': 17848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7133, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2501}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7158", 'reported_seq': 7329, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.083177+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:29:54.083177+0000', 'last_peered': '2024-04-26T02:29:54.083177+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:29:54.083177+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7158, 'log_dups_size': 0, 'ondisk_log_size': 7158, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36388864, 'num_objects': 4442, 'num_object_clones': 0, 'num_object_copies': 17768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7158, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7441", 'reported_seq': 7642, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:56.674521+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:29:56.674521+0000', 'last_peered': '2024-04-26T02:29:56.674521+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:29:56.674521+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7441, 'log_dups_size': 0, 'ondisk_log_size': 7441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37253644, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 18192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7439, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7201", 'reported_seq': 7372, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.094434+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:29:54.094434+0000', 'last_peered': '2024-04-26T02:29:54.094434+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:29:54.094434+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7201, 'log_dups_size': 0, 'ondisk_log_size': 7201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37176344, 'num_objects': 4539, 'num_object_clones': 0, 'num_object_copies': 18156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1952, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2590}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7120", 'reported_seq': 7246, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.510708+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:29:54.510708+0000', 'last_peered': '2024-04-26T02:29:54.510708+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:29:54.510708+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7120, 'log_dups_size': 0, 'ondisk_log_size': 7120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36401676, 'num_objects': 4444, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7118, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2429}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:59.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7183", 'reported_seq': 7338, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.011434+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:29:54.011434+0000', 'last_peered': '2024-04-26T02:29:54.011434+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:29:54.011434+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7183, 'log_dups_size': 0, 'ondisk_log_size': 7183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37153554, 'num_objects': 4536, 'num_object_clones': 0, 'num_object_copies': 18144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1974, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7180, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2565}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7116", 'reported_seq': 7266, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.098117+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:29:54.098117+0000', 'last_peered': '2024-04-26T02:29:54.098117+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:29:54.098117+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7116, 'log_dups_size': 0, 'ondisk_log_size': 7116, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36706566, 'num_objects': 4481, 'num_object_clones': 0, 'num_object_copies': 17924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1936, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7115, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7266", 'reported_seq': 7446, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:54.579416+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:29:54.579416+0000', 'last_peered': '2024-04-26T02:29:54.579416+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:29:54.579416+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7266, 'log_dups_size': 0, 'ondisk_log_size': 7266, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36968710, 'num_objects': 4513, 'num_object_clones': 0, 'num_object_copies': 18052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7265, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7426", 'reported_seq': 7579, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:53.992626+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:29:53.992626+0000', 'last_peered': '2024-04-26T02:29:53.992626+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:29:53.992626+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7426, 'log_dups_size': 0, 'ondisk_log_size': 7426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38466060, 'num_objects': 4696, 'num_object_clones': 0, 'num_object_copies': 18784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7424, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:29:59.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7315", 'reported_seq': 7481, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:29:56.898824+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:29:56.898824+0000', 'last_peered': '2024-04-26T02:29:56.898824+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:29:56.898824+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7315, 'log_dups_size': 0, 'ondisk_log_size': 7315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36712972, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 17928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2018, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7313, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:29:59.280 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:00.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:00.950 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26910/285160 objects degraded (9.437%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":397,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71290,"data_bytes":583949504,"bytes_used":2747707392,"bytes_avail":383799349248,"bytes_total":386547056640,"degraded_objects":26910,"degraded_total":285160,"degraded_ratio":0.094368074063683541,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":139183,"recovering_keys_per_sec":0,"num_objects_recovered":281,"num_bytes_recovered":1671168,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":249},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-26T02:27:30.505375+0000","services":{}},"progress_events":{}} 2024-04-26T02:30:01.043 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:01.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:02.549 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:02.550 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:02.637 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-04-26T02:30:00.584189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583949504,"num_objects":71290,"num_object_clones":0,"num_object_copies":285160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116267,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5859,"num_bytes_recovered":42237952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116294,"ondisk_log_size":116294,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2683308,"kb_used_data":1109168,"kb_used_omap":14,"kb_used_meta":1574065,"kb_avail":374804052,"statfs":{"total":386547056640,"available":383799349248,"internally_reserved":0,"allocated":1135788032,"data_stored":1135010612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1611843123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,1,10,13,10,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24584192,"num_objects":-3001,"num_object_clones":0,"num_object_copies":-12004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006969"},"pg_stats":[{"pgid":"1.f","version":"36'7126","reported_seq":7284,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.055327+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:29:59.055327+0000","last_peered":"2024-04-26T02:29:59.055327+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:29:59.055327+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7126,"log_dups_size":0,"ondisk_log_size":7126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7298","reported_seq":7457,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.875477+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:29:56.875477+0000","last_peered":"2024-04-26T02:29:56.875477+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:29:56.875477+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7298,"log_dups_size":0,"ondisk_log_size":7298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37138956,"num_objects":4534,"num_object_clones":0,"num_object_copies":18136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7202","reported_seq":7346,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.491681+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:29:59.491681+0000","last_peered":"2024-04-26T02:29:59.491681+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:29:59.491681+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7202,"log_dups_size":0,"ondisk_log_size":7202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36272414,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7201,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2435}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7092","reported_seq":9243,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.938862+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:29:56.938862+0000","last_peered":"2024-04-26T02:29:56.938862+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:29:56.938862+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7092,"log_dups_size":0,"ondisk_log_size":7092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35942924,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":8364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7206","reported_seq":7369,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:58.885174+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:29:58.885174+0000","last_peered":"2024-04-26T02:29:58.885174+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:29:58.885174+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7206,"log_dups_size":0,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35967500,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7225","reported_seq":7404,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:58.983301+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:29:58.983301+0000","last_peered":"2024-04-26T02:29:58.983301+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:29:58.983301+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7225,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7279","reported_seq":11526,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:56.826388+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:29:56.826388+0000","last_peered":"2024-04-26T02:29:56.826388+0000","last_clean":"2024-04-26T02:29:56.826388+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:29:56.826388+0000","last_undegraded":"2024-04-26T02:29:56.826388+0000","last_fullsized":"2024-04-26T02:29:56.826388+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:57.947922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36688396,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7277,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7441","reported_seq":7642,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.674521+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:29:56.674521+0000","last_peered":"2024-04-26T02:29:56.674521+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:29:56.674521+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253644,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7273","reported_seq":7453,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.022812+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:29:59.022812+0000","last_peered":"2024-04-26T02:29:59.022812+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:29:59.022812+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36586520,"num_objects":4467,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7198","reported_seq":7329,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.591375+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:29:59.591375+0000","last_peered":"2024-04-26T02:29:59.591375+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:29:59.591375+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35762700,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7264","reported_seq":7431,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.024474+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:29:59.024474+0000","last_peered":"2024-04-26T02:29:59.024474+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:29:59.024474+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36490002,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7181","reported_seq":7339,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.060859+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:29:59.060859+0000","last_peered":"2024-04-26T02:29:59.060859+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:29:59.060859+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36174086,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7180,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7337","reported_seq":7517,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.449491+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:29:59.449491+0000","last_peered":"2024-04-26T02:29:59.449491+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:29:59.449491+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7337,"log_dups_size":0,"ondisk_log_size":7337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36387078,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7498","reported_seq":7660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:59.083460+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:29:59.083460+0000","last_peered":"2024-04-26T02:29:59.083460+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:29:59.083460+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876236,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7359","reported_seq":11563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:29:58.137713+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:29:58.137713+0000","last_peered":"2024-04-26T02:29:58.137713+0000","last_clean":"2024-04-26T02:29:58.137713+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:29:58.137713+0000","last_undegraded":"2024-04-26T02:29:58.137713+0000","last_fullsized":"2024-04-26T02:29:58.137713+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36827660,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7315","reported_seq":7481,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:29:56.898824+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:29:56.898824+0000","last_peered":"2024-04-26T02:29:56.898824+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:29:56.898824+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36712972,"num_objects":4482,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583949504,"num_objects":71290,"num_object_clones":0,"num_object_copies":285160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116267,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5859,"num_bytes_recovered":42237952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1132122112,"data_stored":1132122112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116294,"ondisk_log_size":116294,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718588,"kb_used_data":285672,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93653252,"statfs":{"total":96636764160,"available":95900930048,"internally_reserved":0,"allocated":292528128,"data_stored":292333773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,3,4,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713428,"kb_used_data":285192,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93658412,"statfs":{"total":96636764160,"available":95906213888,"internally_reserved":0,"allocated":292036608,"data_stored":291842253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557068,"kb_used_data":250560,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93814772,"statfs":{"total":96636764160,"available":96066326528,"internally_reserved":0,"allocated":256573440,"data_stored":256379085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694224,"kb_used_data":287744,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93677616,"statfs":{"total":96636764160,"available":95925878784,"internally_reserved":0,"allocated":294649856,"data_stored":294455501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293478400,"data_stored":293478400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256249856,"data_stored":256249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290988032,"data_stored":290988032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291405824,"data_stored":291405824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:05.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:06.830 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:06.831 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:06.936 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-04-26T02:30:06.585712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574774464,"num_objects":70170,"num_object_clones":0,"num_object_copies":280680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5986,"num_bytes_recovered":42893312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117414,"ondisk_log_size":117414,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698768,"kb_used_data":1094116,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374788592,"statfs":{"total":386547056640,"available":383783518208,"internally_reserved":0,"allocated":1120374784,"data_stored":1119597364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1643103795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,9,7,6,18,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20144128,"num_objects":-2459,"num_object_clones":0,"num_object_copies":-9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006956"},"pg_stats":[{"pgid":"1.f","version":"36'7184","reported_seq":7345,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:03.896341+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:03.896341+0000","last_peered":"2024-04-26T02:30:03.896341+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:03.896341+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7184,"log_dups_size":0,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35553280,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7367","reported_seq":7527,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.969440+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:01.969440+0000","last_peered":"2024-04-26T02:30:01.969440+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:01.969440+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36573708,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7365,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7279","reported_seq":7433,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.494441+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:04.494441+0000","last_peered":"2024-04-26T02:30:04.494441+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:04.494441+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35641630,"num_objects":4352,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7278,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7152","reported_seq":9461,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.994326+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:01.994326+0000","last_peered":"2024-04-26T02:30:01.994326+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:01.994326+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35451404,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1123,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3479}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7273","reported_seq":7441,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:03.812146+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:03.812146+0000","last_peered":"2024-04-26T02:30:03.812146+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:03.812146+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35418636,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7296","reported_seq":7498,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.183292+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:04.183292+0000","last_peered":"2024-04-26T02:30:04.183292+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:04.183292+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7345","reported_seq":11592,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:01.948032+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:01.948032+0000","last_peered":"2024-04-26T02:30:01.948032+0000","last_clean":"2024-04-26T02:30:01.948032+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:01.948032+0000","last_undegraded":"2024-04-26T02:30:01.948032+0000","last_fullsized":"2024-04-26T02:30:01.948032+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:03.957901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36147724,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7512","reported_seq":7715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.837893+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:01.837893+0000","last_peered":"2024-04-26T02:30:01.837893+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:01.837893+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672012,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7349","reported_seq":7541,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.018391+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:04.018391+0000","last_peered":"2024-04-26T02:30:04.018391+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:04.018391+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7349,"log_dups_size":0,"ondisk_log_size":7349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35963928,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7277","reported_seq":7412,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.470509+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:04.470509+0000","last_peered":"2024-04-26T02:30:04.470509+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:04.470509+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35115532,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2275}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7341","reported_seq":7515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.190986+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:04.190986+0000","last_peered":"2024-04-26T02:30:04.190986+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:04.190986+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35859218,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7246","reported_seq":7414,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.140841+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:04.140841+0000","last_peered":"2024-04-26T02:30:04.140841+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:04.140841+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35641606,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2425}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7417","reported_seq":7601,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.529587+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:04.529587+0000","last_peered":"2024-04-26T02:30:04.529587+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:04.529587+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35731718,"num_objects":4362,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7568","reported_seq":7738,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.077939+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:04.077939+0000","last_peered":"2024-04-26T02:30:04.077939+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:04.077939+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37302796,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7423","reported_seq":11628,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:03.054972+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:03.054972+0000","last_peered":"2024-04-26T02:30:03.054972+0000","last_clean":"2024-04-26T02:30:03.054972+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:03.054972+0000","last_undegraded":"2024-04-26T02:30:03.054972+0000","last_fullsized":"2024-04-26T02:30:03.054972+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:03.528279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36303372,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7385","reported_seq":7553,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.839599+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:01.839599+0000","last_peered":"2024-04-26T02:30:01.839599+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:01.839599+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36139532,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":574774464,"num_objects":70170,"num_object_clones":0,"num_object_copies":280680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5986,"num_bytes_recovered":42893312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1117437952,"data_stored":1117437952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117414,"ondisk_log_size":117414,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705720,"kb_used_data":281132,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93666120,"statfs":{"total":96636764160,"available":95914106880,"internally_reserved":0,"allocated":287879168,"data_stored":287684813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4,12,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717040,"kb_used_data":280488,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93654800,"statfs":{"total":96636764160,"available":95902515200,"internally_reserved":0,"allocated":287219712,"data_stored":287025357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560748,"kb_used_data":250084,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93811092,"statfs":{"total":96636764160,"available":96062558208,"internally_reserved":0,"allocated":256086016,"data_stored":255891661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715260,"kb_used_data":282412,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93656580,"statfs":{"total":96636764160,"available":95904337920,"internally_reserved":0,"allocated":289189888,"data_stored":288995533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,3,1,2,2,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288784384,"data_stored":288784384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255688704,"data_stored":255688704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286269440,"data_stored":286269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286695424,"data_stored":286695424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:06.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:08.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:08.037 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:08.088 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-04-26T02:30:06.585712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574774464,"num_objects":70170,"num_object_clones":0,"num_object_copies":280680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5986,"num_bytes_recovered":42893312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117414,"ondisk_log_size":117414,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698768,"kb_used_data":1094116,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374788592,"statfs":{"total":386547056640,"available":383783518208,"internally_reserved":0,"allocated":1120374784,"data_stored":1119597364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1643103795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,9,7,6,18,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20144128,"num_objects":-2459,"num_object_clones":0,"num_object_copies":-9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006956"},"pg_stats":[{"pgid":"1.f","version":"36'7184","reported_seq":7345,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:03.896341+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:03.896341+0000","last_peered":"2024-04-26T02:30:03.896341+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:03.896341+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7184,"log_dups_size":0,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35553280,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7367","reported_seq":7527,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.969440+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:01.969440+0000","last_peered":"2024-04-26T02:30:01.969440+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:01.969440+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36573708,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7365,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7279","reported_seq":7433,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.494441+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:04.494441+0000","last_peered":"2024-04-26T02:30:04.494441+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:04.494441+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35641630,"num_objects":4352,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7278,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7152","reported_seq":9461,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.994326+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:01.994326+0000","last_peered":"2024-04-26T02:30:01.994326+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:01.994326+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35451404,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1123,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3479}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7273","reported_seq":7441,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:03.812146+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:03.812146+0000","last_peered":"2024-04-26T02:30:03.812146+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:03.812146+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35418636,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7296","reported_seq":7498,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.183292+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:04.183292+0000","last_peered":"2024-04-26T02:30:04.183292+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:04.183292+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7296,"log_dups_size":0,"ondisk_log_size":7296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7296,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7345","reported_seq":11592,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:01.948032+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:01.948032+0000","last_peered":"2024-04-26T02:30:01.948032+0000","last_clean":"2024-04-26T02:30:01.948032+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:01.948032+0000","last_undegraded":"2024-04-26T02:30:01.948032+0000","last_fullsized":"2024-04-26T02:30:01.948032+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:03.957901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36147724,"num_objects":4413,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7512","reported_seq":7715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.837893+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:01.837893+0000","last_peered":"2024-04-26T02:30:01.837893+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:01.837893+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672012,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7349","reported_seq":7541,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.018391+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:04.018391+0000","last_peered":"2024-04-26T02:30:04.018391+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:04.018391+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7349,"log_dups_size":0,"ondisk_log_size":7349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35963928,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7277","reported_seq":7412,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.470509+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:04.470509+0000","last_peered":"2024-04-26T02:30:04.470509+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:04.470509+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35115532,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2275}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7341","reported_seq":7515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.190986+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:04.190986+0000","last_peered":"2024-04-26T02:30:04.190986+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:04.190986+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7341,"log_dups_size":0,"ondisk_log_size":7341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35859218,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7246","reported_seq":7414,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.140841+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:04.140841+0000","last_peered":"2024-04-26T02:30:04.140841+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:04.140841+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35641606,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2425}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7417","reported_seq":7601,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.529587+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:04.529587+0000","last_peered":"2024-04-26T02:30:04.529587+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:04.529587+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35731718,"num_objects":4362,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7568","reported_seq":7738,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:04.077939+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:04.077939+0000","last_peered":"2024-04-26T02:30:04.077939+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:04.077939+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37302796,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7423","reported_seq":11628,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:03.054972+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:03.054972+0000","last_peered":"2024-04-26T02:30:03.054972+0000","last_clean":"2024-04-26T02:30:03.054972+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:03.054972+0000","last_undegraded":"2024-04-26T02:30:03.054972+0000","last_fullsized":"2024-04-26T02:30:03.054972+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:03.528279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36303372,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7385","reported_seq":7553,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:01.839599+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:01.839599+0000","last_peered":"2024-04-26T02:30:01.839599+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:01.839599+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36139532,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":574774464,"num_objects":70170,"num_object_clones":0,"num_object_copies":280680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26783,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5986,"num_bytes_recovered":42893312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1117437952,"data_stored":1117437952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117414,"ondisk_log_size":117414,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705720,"kb_used_data":281132,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93666120,"statfs":{"total":96636764160,"available":95914106880,"internally_reserved":0,"allocated":287879168,"data_stored":287684813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4,12,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717040,"kb_used_data":280488,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93654800,"statfs":{"total":96636764160,"available":95902515200,"internally_reserved":0,"allocated":287219712,"data_stored":287025357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560748,"kb_used_data":250084,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93811092,"statfs":{"total":96636764160,"available":96062558208,"internally_reserved":0,"allocated":256086016,"data_stored":255891661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715260,"kb_used_data":282412,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93656580,"statfs":{"total":96636764160,"available":95904337920,"internally_reserved":0,"allocated":289189888,"data_stored":288995533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,3,1,2,2,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288784384,"data_stored":288784384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255688704,"data_stored":255688704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286269440,"data_stored":286269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286695424,"data_stored":286695424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:08.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:08.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7184", 'reported_seq': 7345, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:03.896341+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:03.896341+0000', 'last_peered': '2024-04-26T02:30:03.896341+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:03.896341+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7184, 'log_dups_size': 0, 'ondisk_log_size': 7184, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35553280, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7184, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2380}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:08.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7367", 'reported_seq': 7527, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:01.969440+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:01.969440+0000', 'last_peered': '2024-04-26T02:30:01.969440+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:01.969440+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7367, 'log_dups_size': 0, 'ondisk_log_size': 7367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36573708, 'num_objects': 4465, 'num_object_clones': 0, 'num_object_copies': 17860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7365, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7279", 'reported_seq': 7433, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.494441+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:04.494441+0000', 'last_peered': '2024-04-26T02:30:04.494441+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:04.494441+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7279, 'log_dups_size': 0, 'ondisk_log_size': 7279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35641630, 'num_objects': 4352, 'num_object_clones': 0, 'num_object_copies': 17408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4352, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7278, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2363}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'7152", 'reported_seq': 9461, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:01.994326+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:30:01.994326+0000', 'last_peered': '2024-04-26T02:30:01.994326+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:30:01.994326+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7152, 'log_dups_size': 0, 'ondisk_log_size': 7152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35451404, 'num_objects': 4328, 'num_object_clones': 0, 'num_object_copies': 17312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 849, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7150, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1123, 'num_bytes_recovered': 9019392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3479}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7273", 'reported_seq': 7441, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:03.812146+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:03.812146+0000', 'last_peered': '2024-04-26T02:30:03.812146+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:03.812146+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7273, 'log_dups_size': 0, 'ondisk_log_size': 7273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35418636, 'num_objects': 4324, 'num_object_clones': 0, 'num_object_copies': 17296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7271, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7296", 'reported_seq': 7498, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.183292+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:04.183292+0000', 'last_peered': '2024-04-26T02:30:04.183292+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:04.183292+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7296, 'log_dups_size': 0, 'ondisk_log_size': 7296, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35258368, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1974, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7296, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:08.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7512", 'reported_seq': 7715, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:01.837893+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:01.837893+0000', 'last_peered': '2024-04-26T02:30:01.837893+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:01.837893+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7512, 'log_dups_size': 0, 'ondisk_log_size': 7512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36672012, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7510, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7349", 'reported_seq': 7541, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.018391+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:30:04.018391+0000', 'last_peered': '2024-04-26T02:30:04.018391+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:04.018391+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7349, 'log_dups_size': 0, 'ondisk_log_size': 7349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35963928, 'num_objects': 4391, 'num_object_clones': 0, 'num_object_copies': 17564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1942, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7346, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7277", 'reported_seq': 7412, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.470509+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:04.470509+0000', 'last_peered': '2024-04-26T02:30:04.470509+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:04.470509+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7277, 'log_dups_size': 0, 'ondisk_log_size': 7277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35115532, 'num_objects': 4287, 'num_object_clones': 0, 'num_object_copies': 17148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7275, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2275}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7341", 'reported_seq': 7515, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.190986+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:04.190986+0000', 'last_peered': '2024-04-26T02:30:04.190986+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:04.190986+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7341, 'log_dups_size': 0, 'ondisk_log_size': 7341, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35859218, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7338, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2417}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7246", 'reported_seq': 7414, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.140841+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:04.140841+0000', 'last_peered': '2024-04-26T02:30:04.140841+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:04.140841+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7246, 'log_dups_size': 0, 'ondisk_log_size': 7246, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35641606, 'num_objects': 4351, 'num_object_clones': 0, 'num_object_copies': 17404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7245, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2425}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7417", 'reported_seq': 7601, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.529587+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:04.529587+0000', 'last_peered': '2024-04-26T02:30:04.529587+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:04.529587+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7417, 'log_dups_size': 0, 'ondisk_log_size': 7417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35731718, 'num_objects': 4362, 'num_object_clones': 0, 'num_object_copies': 17448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7416, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:08.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:08.092 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7568", 'reported_seq': 7738, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:04.077939+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:04.077939+0000', 'last_peered': '2024-04-26T02:30:04.077939+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:04.077939+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7568, 'log_dups_size': 0, 'ondisk_log_size': 7568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37302796, 'num_objects': 4554, 'num_object_clones': 0, 'num_object_copies': 18216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7566, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:08.092 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:08.092 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7385", 'reported_seq': 7553, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:01.839599+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:01.839599+0000', 'last_peered': '2024-04-26T02:30:01.839599+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:01.839599+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7385, 'log_dups_size': 0, 'ondisk_log_size': 7385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36139532, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7383, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:08.092 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:09.761 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:09.761 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26697/278940 objects degraded (9.571%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":406,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69735,"data_bytes":571210944,"bytes_used":2754052096,"bytes_avail":383793004544,"bytes_total":386547056640,"degraded_objects":26697,"degraded_total":278940,"degraded_ratio":0.095708754570875451,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":189671,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":2277376,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-26T02:30:06.585774+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:09.840 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:09.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:11.500 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:11.501 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:11.580 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-26T02:30:10.588766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":565583040,"num_objects":69048,"num_object_clones":0,"num_object_copies":276192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118509,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6087,"num_bytes_recovered":43515904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118536,"ondisk_log_size":118536,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672364,"kb_used_data":1080200,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374814996,"statfs":{"total":386547056640,"available":383810555904,"internally_reserved":0,"allocated":1106124800,"data_stored":1105347380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1630324275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,5,4,9,1,0,0,4,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24420352,"num_objects":-2981,"num_object_clones":0,"num_object_copies":-11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008999"},"pg_stats":[{"pgid":"1.f","version":"36'7255","reported_seq":7416,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.144374+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:09.144374+0000","last_peered":"2024-04-26T02:30:09.144374+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:09.144374+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7255,"log_dups_size":0,"ondisk_log_size":7255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34971648,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7255,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7439","reported_seq":7610,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:06.941849+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:06.941849+0000","last_peered":"2024-04-26T02:30:06.941849+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:06.941849+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7439,"log_dups_size":0,"ondisk_log_size":7439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35983884,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7437,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7360","reported_seq":7515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.676231+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:09.676231+0000","last_peered":"2024-04-26T02:30:09.676231+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:09.676231+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34978078,"num_objects":4271,"num_object_clones":0,"num_object_copies":17084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7359,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7235","reported_seq":9697,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:07.046714+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:07.046714+0000","last_peered":"2024-04-26T02:30:07.046714+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:07.046714+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771468,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":9641984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7330","reported_seq":7507,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.026568+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:09.026568+0000","last_peered":"2024-04-26T02:30:09.026568+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:09.026568+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34951692,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7373","reported_seq":7582,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.251798+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:09.251798+0000","last_peered":"2024-04-26T02:30:09.251798+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:09.251798+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7413","reported_seq":11660,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:06.973311+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:06.973311+0000","last_peered":"2024-04-26T02:30:06.973311+0000","last_clean":"2024-04-26T02:30:06.973311+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:06.973311+0000","last_undegraded":"2024-04-26T02:30:06.973311+0000","last_fullsized":"2024-04-26T02:30:06.973311+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:09.891785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35590668,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7411,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7591","reported_seq":7799,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:06.963156+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:06.963156+0000","last_peered":"2024-04-26T02:30:06.963156+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:06.963156+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7591,"log_dups_size":0,"ondisk_log_size":7591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36024844,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7589,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7413","reported_seq":7607,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:08.987128+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:08.987128+0000","last_peered":"2024-04-26T02:30:08.987128+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:08.987128+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35439640,"num_objects":4327,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7410,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7355","reported_seq":7490,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.482100+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:09.482100+0000","last_peered":"2024-04-26T02:30:09.482100+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:09.482100+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34476556,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2197}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7406","reported_seq":7581,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.139270+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:09.139270+0000","last_peered":"2024-04-26T02:30:09.139270+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:09.139270+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326738,"num_objects":4313,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7316","reported_seq":7486,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.063649+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:09.063649+0000","last_peered":"2024-04-26T02:30:09.063649+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:09.063649+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068166,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7479","reported_seq":7665,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.485393+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:09.485393+0000","last_peered":"2024-04-26T02:30:09.485393+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:09.485393+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35223814,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7630","reported_seq":7807,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:08.864568+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:08.864568+0000","last_peered":"2024-04-26T02:30:08.864568+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:08.864568+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7630,"log_dups_size":0,"ondisk_log_size":7630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36794892,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7491","reported_seq":11696,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:08.117757+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:08.117757+0000","last_peered":"2024-04-26T02:30:08.117757+0000","last_clean":"2024-04-26T02:30:08.117757+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:08.117757+0000","last_undegraded":"2024-04-26T02:30:08.117757+0000","last_fullsized":"2024-04-26T02:30:08.117757+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:09.507283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35746316,"num_objects":4364,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7489,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7450","reported_seq":7622,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:06.854038+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:06.854038+0000","last_peered":"2024-04-26T02:30:06.854038+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:06.854038+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7450,"log_dups_size":0,"ondisk_log_size":7450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35607052,"num_objects":4347,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7448,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":565583040,"num_objects":69048,"num_object_clones":0,"num_object_copies":276192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118509,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6087,"num_bytes_recovered":43515904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1103024128,"data_stored":1103024128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118536,"ondisk_log_size":118536,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705372,"kb_used_data":276628,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93666468,"statfs":{"total":96636764160,"available":95914463232,"internally_reserved":0,"allocated":283267072,"data_stored":283072717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,0,4,1,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700248,"kb_used_data":276176,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93671592,"statfs":{"total":96636764160,"available":95919710208,"internally_reserved":0,"allocated":282804224,"data_stored":282609869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551836,"kb_used_data":249488,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93820004,"statfs":{"total":96636764160,"available":96071684096,"internally_reserved":0,"allocated":255475712,"data_stored":255281357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714908,"kb_used_data":277908,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93656932,"statfs":{"total":96636764160,"available":95904698368,"internally_reserved":0,"allocated":284577792,"data_stored":284383437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,2,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284094464,"data_stored":284094464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255119360,"data_stored":255119360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281710592,"data_stored":281710592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":282099712,"data_stored":282099712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:14.582 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:16.104 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:16.171 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-26T02:30:14.593158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558095552,"num_objects":68134,"num_object_clones":0,"num_object_copies":272536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119423,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6190,"num_bytes_recovered":44204032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119450,"ondisk_log_size":119450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666332,"kb_used_data":1070012,"kb_used_omap":14,"kb_used_meta":1596273,"kb_avail":374821028,"statfs":{"total":386547056640,"available":383816732672,"internally_reserved":0,"allocated":1095692288,"data_stored":1094914868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1634584115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,4,13,15,11,2,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23101440,"num_objects":-2820,"num_object_clones":0,"num_object_copies":-11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2820,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008337"},"pg_stats":[{"pgid":"1.f","version":"36'7325","reported_seq":7500,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.143270+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:14.143270+0000","last_peered":"2024-04-26T02:30:14.143270+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:14.143270+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34398208,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7528","reported_seq":7705,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.073969+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:12.073969+0000","last_peered":"2024-04-26T02:30:12.073969+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:12.073969+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35254796,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7526,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7360","reported_seq":7515,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.676231+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:09.676231+0000","last_peered":"2024-04-26T02:30:09.676231+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:09.676231+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34978078,"num_objects":4271,"num_object_clones":0,"num_object_copies":17084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7359,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2283}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7308","reported_seq":9939,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.099492+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:12.099492+0000","last_peered":"2024-04-26T02:30:12.099492+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:12.099492+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34173452,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7306,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":10330112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7395","reported_seq":7576,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.192432+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:14.192432+0000","last_peered":"2024-04-26T02:30:14.192432+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:14.192432+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419212,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7393,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7435","reported_seq":7648,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.037696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:14.037696+0000","last_peered":"2024-04-26T02:30:14.037696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:14.037696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7486","reported_seq":11733,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:12.010996+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:12.010996+0000","last_peered":"2024-04-26T02:30:12.010996+0000","last_clean":"2024-04-26T02:30:12.010996+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:12.010996+0000","last_undegraded":"2024-04-26T02:30:12.010996+0000","last_fullsized":"2024-04-26T02:30:12.010996+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:15.796739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34992652,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7667","reported_seq":7877,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.050509+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:12.050509+0000","last_peered":"2024-04-26T02:30:12.050509+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:12.050509+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35402252,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7478","reported_seq":7676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.138263+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:14.138263+0000","last_peered":"2024-04-26T02:30:14.138263+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:14.138263+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34907160,"num_objects":4262,"num_object_clones":0,"num_object_copies":17048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7475,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7355","reported_seq":7490,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.482100+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:09.482100+0000","last_peered":"2024-04-26T02:30:09.482100+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:09.482100+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34476556,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2197}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7483","reported_seq":7660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.203492+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:14.203492+0000","last_peered":"2024-04-26T02:30:14.203492+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:14.203492+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34695954,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7393","reported_seq":7565,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.272352+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:14.272352+0000","last_peered":"2024-04-26T02:30:14.272352+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:14.272352+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34437382,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7479","reported_seq":7665,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:09.485393+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:09.485393+0000","last_peered":"2024-04-26T02:30:09.485393+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:09.485393+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35223814,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7693","reported_seq":7871,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.163759+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:14.163759+0000","last_peered":"2024-04-26T02:30:14.163759+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:14.163759+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36278796,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7553","reported_seq":11758,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:12.960948+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:12.960948+0000","last_peered":"2024-04-26T02:30:12.960948+0000","last_clean":"2024-04-26T02:30:12.960948+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:12.960948+0000","last_undegraded":"2024-04-26T02:30:12.960948+0000","last_fullsized":"2024-04-26T02:30:12.960948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:14.571807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35238412,"num_objects":4302,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7512","reported_seq":7684,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.033881+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:12.033881+0000","last_peered":"2024-04-26T02:30:12.033881+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:12.033881+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35099148,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558095552,"num_objects":68134,"num_object_clones":0,"num_object_copies":272536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119423,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6190,"num_bytes_recovered":44204032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1093160960,"data_stored":1093160960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119450,"ondisk_log_size":119450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":271252,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":277762048,"data_stored":277567693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,2,7,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700248,"kb_used_data":276176,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93671592,"statfs":{"total":96636764160,"available":95919710208,"internally_reserved":0,"allocated":282804224,"data_stored":282609869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555540,"kb_used_data":249036,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93816300,"statfs":{"total":96636764160,"available":96067891200,"internally_reserved":0,"allocated":255012864,"data_stored":254818509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702228,"kb_used_data":273548,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93669612,"statfs":{"total":96636764160,"available":95917682688,"internally_reserved":0,"allocated":280113152,"data_stored":279918797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,5,3,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279453696,"data_stored":279453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254603264,"data_stored":254603264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":281710592,"data_stored":281710592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277393408,"data_stored":277393408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:16.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:17.535 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:17.535 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:17.612 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-04-26T02:30:16.596233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":556195008,"num_objects":67902,"num_object_clones":0,"num_object_copies":271608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119655,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6190,"num_bytes_recovered":44204032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119682,"ondisk_log_size":119682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665872,"kb_used_data":1065392,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374821488,"statfs":{"total":386547056640,"available":383817203712,"internally_reserved":0,"allocated":1090961408,"data_stored":1090183988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1638843955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,8,12,17,11,4,0,0,0,1,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20512768,"num_objects":-2504,"num_object_clones":0,"num_object_copies":-10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010914"},"pg_stats":[{"pgid":"1.f","version":"36'7325","reported_seq":7500,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.143270+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:14.143270+0000","last_peered":"2024-04-26T02:30:14.143270+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:14.143270+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34398208,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7528","reported_seq":7705,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.073969+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:12.073969+0000","last_peered":"2024-04-26T02:30:12.073969+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:12.073969+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35254796,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7526,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7431","reported_seq":7587,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.440974+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:14.440974+0000","last_peered":"2024-04-26T02:30:14.440974+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:14.440974+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7431,"log_dups_size":0,"ondisk_log_size":7431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34396446,"num_objects":4200,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7430,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2212}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7308","reported_seq":9939,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.099492+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:12.099492+0000","last_peered":"2024-04-26T02:30:12.099492+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:12.099492+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34173452,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7306,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":10330112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7395","reported_seq":7576,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.192432+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:14.192432+0000","last_peered":"2024-04-26T02:30:14.192432+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:14.192432+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34419212,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7393,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7435","reported_seq":7648,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.037696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:14.037696+0000","last_peered":"2024-04-26T02:30:14.037696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:14.037696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":4165,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7486","reported_seq":11733,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:12.010996+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:12.010996+0000","last_peered":"2024-04-26T02:30:12.010996+0000","last_clean":"2024-04-26T02:30:12.010996+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:12.010996+0000","last_undegraded":"2024-04-26T02:30:12.010996+0000","last_fullsized":"2024-04-26T02:30:12.010996+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:15.796739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34992652,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7484,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7667","reported_seq":7877,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.050509+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:12.050509+0000","last_peered":"2024-04-26T02:30:12.050509+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:12.050509+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35402252,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7478","reported_seq":7676,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.138263+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:14.138263+0000","last_peered":"2024-04-26T02:30:14.138263+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:14.138263+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34907160,"num_objects":4262,"num_object_clones":0,"num_object_copies":17048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7475,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7429","reported_seq":7564,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.606764+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:14.606764+0000","last_peered":"2024-04-26T02:30:14.606764+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:14.606764+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33870348,"num_objects":4135,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7483","reported_seq":7660,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.203492+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:14.203492+0000","last_peered":"2024-04-26T02:30:14.203492+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:14.203492+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34695954,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7393","reported_seq":7565,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.272352+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:14.272352+0000","last_peered":"2024-04-26T02:30:14.272352+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:14.272352+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34437382,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7566","reported_seq":7752,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.634575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:14.634575+0000","last_peered":"2024-04-26T02:30:14.634575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:14.634575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34511110,"num_objects":4213,"num_object_clones":0,"num_object_copies":16852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2155}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7693","reported_seq":7871,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:14.163759+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:14.163759+0000","last_peered":"2024-04-26T02:30:14.163759+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:14.163759+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36278796,"num_objects":4429,"num_object_clones":0,"num_object_copies":17716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7553","reported_seq":11758,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:12.960948+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:12.960948+0000","last_peered":"2024-04-26T02:30:12.960948+0000","last_clean":"2024-04-26T02:30:12.960948+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:12.960948+0000","last_undegraded":"2024-04-26T02:30:12.960948+0000","last_fullsized":"2024-04-26T02:30:12.960948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:14.571807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35238412,"num_objects":4302,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7512","reported_seq":7684,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:12.033881+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:12.033881+0000","last_peered":"2024-04-26T02:30:12.033881+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:12.033881+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35099148,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":556195008,"num_objects":67902,"num_object_clones":0,"num_object_copies":271608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119655,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6190,"num_bytes_recovered":44204032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088520192,"data_stored":1088520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119682,"ondisk_log_size":119682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":271252,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":277762048,"data_stored":277567693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,2,7,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699788,"kb_used_data":271556,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93672052,"statfs":{"total":96636764160,"available":95920181248,"internally_reserved":0,"allocated":278073344,"data_stored":277878989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,0,3,2,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555540,"kb_used_data":249036,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93816300,"statfs":{"total":96636764160,"available":96067891200,"internally_reserved":0,"allocated":255012864,"data_stored":254818509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702228,"kb_used_data":273548,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93669612,"statfs":{"total":96636764160,"available":95917682688,"internally_reserved":0,"allocated":280113152,"data_stored":279918797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,5,3,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279453696,"data_stored":279453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254603264,"data_stored":254603264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277069824,"data_stored":277069824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277393408,"data_stored":277393408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7325", 'reported_seq': 7500, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.143270+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:14.143270+0000', 'last_peered': '2024-04-26T02:30:14.143270+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:14.143270+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7325, 'log_dups_size': 0, 'ondisk_log_size': 7325, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34398208, 'num_objects': 4199, 'num_object_clones': 0, 'num_object_copies': 16796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7325, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 64, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7528", 'reported_seq': 7705, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:12.073969+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:12.073969+0000', 'last_peered': '2024-04-26T02:30:12.073969+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:12.073969+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7528, 'log_dups_size': 0, 'ondisk_log_size': 7528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35254796, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7526, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7431", 'reported_seq': 7587, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.440974+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:14.440974+0000', 'last_peered': '2024-04-26T02:30:14.440974+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:14.440974+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7431, 'log_dups_size': 0, 'ondisk_log_size': 7431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34396446, 'num_objects': 4200, 'num_object_clones': 0, 'num_object_copies': 16800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4200, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7430, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2212}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'7308", 'reported_seq': 9939, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:12.099492+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:30:12.099492+0000', 'last_peered': '2024-04-26T02:30:12.099492+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:30:12.099492+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7308, 'log_dups_size': 0, 'ondisk_log_size': 7308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34173452, 'num_objects': 4172, 'num_object_clones': 0, 'num_object_copies': 16688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 689, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7306, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1283, 'num_bytes_recovered': 10330112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7395", 'reported_seq': 7576, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.192432+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:14.192432+0000', 'last_peered': '2024-04-26T02:30:14.192432+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:14.192432+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7395, 'log_dups_size': 0, 'ondisk_log_size': 7395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34419212, 'num_objects': 4202, 'num_object_clones': 0, 'num_object_copies': 16808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7393, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2251}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:17.614 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7435", 'reported_seq': 7648, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.037696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:14.037696+0000', 'last_peered': '2024-04-26T02:30:14.037696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:14.037696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7435, 'log_dups_size': 0, 'ondisk_log_size': 7435, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34119680, 'num_objects': 4165, 'num_object_clones': 0, 'num_object_copies': 16660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7435, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7667", 'reported_seq': 7877, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:12.050509+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:12.050509+0000', 'last_peered': '2024-04-26T02:30:12.050509+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:12.050509+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7667, 'log_dups_size': 0, 'ondisk_log_size': 7667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35402252, 'num_objects': 4322, 'num_object_clones': 0, 'num_object_copies': 17288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7665, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7478", 'reported_seq': 7676, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.138263+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:30:14.138263+0000', 'last_peered': '2024-04-26T02:30:14.138263+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:14.138263+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7478, 'log_dups_size': 0, 'ondisk_log_size': 7478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34907160, 'num_objects': 4262, 'num_object_clones': 0, 'num_object_copies': 17048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7475, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7429", 'reported_seq': 7564, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.606764+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:14.606764+0000', 'last_peered': '2024-04-26T02:30:14.606764+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:14.606764+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7429, 'log_dups_size': 0, 'ondisk_log_size': 7429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33870348, 'num_objects': 4135, 'num_object_clones': 0, 'num_object_copies': 16540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7427, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2123}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7483", 'reported_seq': 7660, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.203492+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:14.203492+0000', 'last_peered': '2024-04-26T02:30:14.203492+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:14.203492+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7483, 'log_dups_size': 0, 'ondisk_log_size': 7483, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34695954, 'num_objects': 4236, 'num_object_clones': 0, 'num_object_copies': 16944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7480, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7393", 'reported_seq': 7565, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.272352+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:14.272352+0000', 'last_peered': '2024-04-26T02:30:14.272352+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:14.272352+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7393, 'log_dups_size': 0, 'ondisk_log_size': 7393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34437382, 'num_objects': 4204, 'num_object_clones': 0, 'num_object_copies': 16816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1925, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7392, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 65, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7566", 'reported_seq': 7752, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.634575+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:14.634575+0000', 'last_peered': '2024-04-26T02:30:14.634575+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:14.634575+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7566, 'log_dups_size': 0, 'ondisk_log_size': 7566, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34511110, 'num_objects': 4213, 'num_object_clones': 0, 'num_object_copies': 16852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2059, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7565, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7693", 'reported_seq': 7871, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:14.163759+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:14.163759+0000', 'last_peered': '2024-04-26T02:30:14.163759+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:14.163759+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7693, 'log_dups_size': 0, 'ondisk_log_size': 7693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36278796, 'num_objects': 4429, 'num_object_clones': 0, 'num_object_copies': 17716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2013, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7691, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:17.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7512", 'reported_seq': 7684, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:12.033881+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:12.033881+0000', 'last_peered': '2024-04-26T02:30:12.033881+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:12.033881+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7512, 'log_dups_size': 0, 'ondisk_log_size': 7512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35099148, 'num_objects': 4285, 'num_object_clones': 0, 'num_object_copies': 17140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7510, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:17.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:19.201 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:19.201 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26491/269956 objects degraded (9.813%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":415,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67489,"data_bytes":552811712,"bytes_used":2737573888,"bytes_avail":383809482752,"bytes_total":386547056640,"degraded_objects":26491,"degraded_total":269956,"degraded_ratio":0.098130806501800297,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":166416,"recovering_keys_per_sec":0,"num_objects_recovered":292,"num_bytes_recovered":1998848,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-26T02:30:06.585774+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:19.244 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:19.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:20.742 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:20.769 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-04-26T02:30:20.597326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":547011776,"num_objects":66781,"num_object_clones":0,"num_object_copies":267124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120776,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6323,"num_bytes_recovered":44892160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120803,"ondisk_log_size":120803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656224,"kb_used_data":1051584,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374831136,"statfs":{"total":386547056640,"available":383827083264,"internally_reserved":0,"allocated":1076822016,"data_stored":1076044596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1643103795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,6,9,5,7,0,0,4,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24199168,"num_objects":-2954,"num_object_clones":0,"num_object_copies":-11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011113"},"pg_stats":[{"pgid":"1.f","version":"36'7404","reported_seq":7599,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.295256+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:19.295256+0000","last_peered":"2024-04-26T02:30:19.295256+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:19.295256+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7404,"log_dups_size":0,"ondisk_log_size":7404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":4120,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7404,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7595","reported_seq":7776,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:17.093432+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:17.093432+0000","last_peered":"2024-04-26T02:30:17.093432+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:17.093432+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34705932,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7501","reported_seq":7661,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.476898+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:19.476898+0000","last_peered":"2024-04-26T02:30:19.476898+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:19.476898+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33823006,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7500,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7364","reported_seq":10164,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:17.164730+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:17.164730+0000","last_peered":"2024-04-26T02:30:17.164730+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:17.164730+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7364,"log_dups_size":0,"ondisk_log_size":7364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33714700,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":11018240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7474","reported_seq":7664,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.348868+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:19.348868+0000","last_peered":"2024-04-26T02:30:19.348868+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:19.348868+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7474,"log_dups_size":0,"ondisk_log_size":7474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772044,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7500","reported_seq":7715,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.283287+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:19.283287+0000","last_peered":"2024-04-26T02:30:19.283287+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:19.283287+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33587200,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7573","reported_seq":11820,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:17.158157+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:17.158157+0000","last_peered":"2024-04-26T02:30:17.158157+0000","last_clean":"2024-04-26T02:30:17.158157+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:17.158157+0000","last_undegraded":"2024-04-26T02:30:17.158157+0000","last_fullsized":"2024-04-26T02:30:17.158157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:21.751193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34279948,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7741","reported_seq":7952,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:17.221086+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:17.221086+0000","last_peered":"2024-04-26T02:30:17.221086+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:17.221086+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796044,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7739,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7549","reported_seq":7755,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.222947+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:19.222947+0000","last_peered":"2024-04-26T02:30:19.222947+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:19.222947+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34325528,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7546,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7496","reported_seq":7637,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.594887+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:19.594887+0000","last_peered":"2024-04-26T02:30:19.594887+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:19.594887+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321484,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7545","reported_seq":7726,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.280484+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:19.280484+0000","last_peered":"2024-04-26T02:30:19.280484+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:19.280484+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7545,"log_dups_size":0,"ondisk_log_size":7545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34188050,"num_objects":4174,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7542,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7457","reported_seq":7641,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.168162+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:19.168162+0000","last_peered":"2024-04-26T02:30:19.168162+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:19.168162+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33913094,"num_objects":4140,"num_object_clones":0,"num_object_copies":16560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7646","reported_seq":7836,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.337774+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:19.337774+0000","last_peered":"2024-04-26T02:30:19.337774+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:19.337774+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33855750,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7764","reported_seq":7962,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.221816+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:19.221816+0000","last_peered":"2024-04-26T02:30:19.221816+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:19.221816+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35697164,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2357}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7614","reported_seq":11819,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:18.261092+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:18.261092+0000","last_peered":"2024-04-26T02:30:18.261092+0000","last_clean":"2024-04-26T02:30:18.261092+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:18.261092+0000","last_undegraded":"2024-04-26T02:30:18.261092+0000","last_fullsized":"2024-04-26T02:30:18.261092+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7614,"log_dups_size":0,"ondisk_log_size":7614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:20.516110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34738700,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7580","reported_seq":7754,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:16.990218+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:16.990218+0000","last_peered":"2024-04-26T02:30:16.990218+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:16.990218+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542092,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":547011776,"num_objects":66781,"num_object_clones":0,"num_object_copies":267124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120776,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6323,"num_bytes_recovered":44892160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1074343936,"data_stored":1074343936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120803,"ondisk_log_size":120803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691384,"kb_used_data":266800,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93680456,"statfs":{"total":96636764160,"available":95928786944,"internally_reserved":0,"allocated":273203200,"data_stored":273008845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,3,1,3,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699532,"kb_used_data":267140,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93672308,"statfs":{"total":96636764160,"available":95920443392,"internally_reserved":0,"allocated":273551360,"data_stored":273357005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563380,"kb_used_data":248556,"kb_used_omap":4,"kb_used_meta":314811,"kb_avail":93808460,"statfs":{"total":96636764160,"available":96059863040,"internally_reserved":0,"allocated":254521344,"data_stored":254326989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":322367183},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701928,"kb_used_data":269088,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93669912,"statfs":{"total":96636764160,"available":95917989888,"internally_reserved":0,"allocated":275546112,"data_stored":275351757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,4,2,2,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274812928,"data_stored":274812928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254164992,"data_stored":254164992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272535552,"data_stored":272535552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272830464,"data_stored":272830464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:23.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:24.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:24.950 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:25.083 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-04-26T02:30:24.598375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539671744,"num_objects":65885,"num_object_clones":0,"num_object_copies":263540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121672,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6484,"num_bytes_recovered":45883392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121699,"ondisk_log_size":121699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643124,"kb_used_data":1042644,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374844236,"statfs":{"total":386547056640,"available":383840497664,"internally_reserved":0,"allocated":1067667456,"data_stored":1066890036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1638843955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,1,8,1,4,6,2,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22855680,"num_objects":-2790,"num_object_clones":0,"num_object_copies":-11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007822"},"pg_stats":[{"pgid":"1.f","version":"36'7470","reported_seq":7673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.414199+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:24.414199+0000","last_peered":"2024-04-26T02:30:24.414199+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:24.414199+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7664","reported_seq":7851,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.144845+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:22.144845+0000","last_peered":"2024-04-26T02:30:22.144845+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:22.144845+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34140684,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7501","reported_seq":7661,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.476898+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:19.476898+0000","last_peered":"2024-04-26T02:30:19.476898+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:19.476898+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33823006,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7500,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7427","reported_seq":10468,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.255325+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:22.255325+0000","last_peered":"2024-04-26T02:30:22.255325+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:22.255325+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33198604,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1488,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7522","reported_seq":7718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.316078+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:24.316078+0000","last_peered":"2024-04-26T02:30:24.316078+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:24.316078+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33378828,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7569","reported_seq":7789,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.401311+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:24.401311+0000","last_peered":"2024-04-26T02:30:24.401311+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:24.401311+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7644","reported_seq":11891,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:22.223882+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:22.223882+0000","last_peered":"2024-04-26T02:30:22.223882+0000","last_clean":"2024-04-26T02:30:22.223882+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:22.223882+0000","last_undegraded":"2024-04-26T02:30:22.223882+0000","last_fullsized":"2024-04-26T02:30:22.223882+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:26.846292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33698316,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7807","reported_seq":8022,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.234807+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:22.234807+0000","last_peered":"2024-04-26T02:30:22.234807+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:22.234807+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34255372,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7623","reported_seq":7835,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.323333+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:24.323333+0000","last_peered":"2024-04-26T02:30:24.323333+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:24.323333+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33719320,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7496","reported_seq":7637,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.594887+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:19.594887+0000","last_peered":"2024-04-26T02:30:19.594887+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:19.594887+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321484,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7618","reported_seq":7807,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.330746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:24.330746+0000","last_peered":"2024-04-26T02:30:24.330746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:24.330746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33590034,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7531","reported_seq":7729,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.397495+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:24.397495+0000","last_peered":"2024-04-26T02:30:24.397495+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:24.397495+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33306886,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7530,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2156}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7646","reported_seq":7836,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.337774+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:19.337774+0000","last_peered":"2024-04-26T02:30:19.337774+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:19.337774+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33855750,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7832","reported_seq":8038,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.229949+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:24.229949+0000","last_peered":"2024-04-26T02:30:24.229949+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:24.229949+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35140108,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7687","reported_seq":11892,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:23.305008+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:23.305008+0000","last_peered":"2024-04-26T02:30:23.305008+0000","last_clean":"2024-04-26T02:30:23.305008+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:23.305008+0000","last_undegraded":"2024-04-26T02:30:23.305008+0000","last_fullsized":"2024-04-26T02:30:23.305008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:26.485045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34140684,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7662","reported_seq":7852,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.144112+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:22.144112+0000","last_peered":"2024-04-26T02:30:22.144112+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:22.144112+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7662,"log_dups_size":0,"ondisk_log_size":7662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33870348,"num_objects":4135,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7660,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539671744,"num_objects":65885,"num_object_clones":0,"num_object_copies":263540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121672,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6484,"num_bytes_recovered":45883392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064988672,"data_stored":1064988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121699,"ondisk_log_size":121699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691368,"kb_used_data":262624,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93680472,"statfs":{"total":96636764160,"available":95928803328,"internally_reserved":0,"allocated":268926976,"data_stored":268732621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,4,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699532,"kb_used_data":267140,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93672308,"statfs":{"total":96636764160,"available":95920443392,"internally_reserved":0,"allocated":273551360,"data_stored":273357005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550580,"kb_used_data":248236,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93821260,"statfs":{"total":96636764160,"available":96072970240,"internally_reserved":0,"allocated":254193664,"data_stored":253999309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701644,"kb_used_data":264644,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93670196,"statfs":{"total":96636764160,"available":95918280704,"internally_reserved":0,"allocated":270995456,"data_stored":270801101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270295040,"data_stored":270295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253743104,"data_stored":253743104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272535552,"data_stored":272535552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268414976,"data_stored":268414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:25.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:26.551 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:26.552 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:26.690 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-04-26T02:30:24.598375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539671744,"num_objects":65885,"num_object_clones":0,"num_object_copies":263540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121672,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6484,"num_bytes_recovered":45883392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121699,"ondisk_log_size":121699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643124,"kb_used_data":1042644,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374844236,"statfs":{"total":386547056640,"available":383840497664,"internally_reserved":0,"allocated":1067667456,"data_stored":1066890036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1638843955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,5,1,8,1,4,6,2,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22855680,"num_objects":-2790,"num_object_clones":0,"num_object_copies":-11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007822"},"pg_stats":[{"pgid":"1.f","version":"36'7470","reported_seq":7673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.414199+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:24.414199+0000","last_peered":"2024-04-26T02:30:24.414199+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:24.414199+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7664","reported_seq":7851,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.144845+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:22.144845+0000","last_peered":"2024-04-26T02:30:22.144845+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:22.144845+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34140684,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7501","reported_seq":7661,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.476898+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:19.476898+0000","last_peered":"2024-04-26T02:30:19.476898+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:19.476898+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33823006,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7500,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2144}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7427","reported_seq":10468,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.255325+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:22.255325+0000","last_peered":"2024-04-26T02:30:22.255325+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:22.255325+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33198604,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7425,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1488,"num_bytes_recovered":12009472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7522","reported_seq":7718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.316078+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:24.316078+0000","last_peered":"2024-04-26T02:30:24.316078+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:24.316078+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33378828,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7569","reported_seq":7789,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.401311+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:24.401311+0000","last_peered":"2024-04-26T02:30:24.401311+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:24.401311+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7644","reported_seq":11891,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:22.223882+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:22.223882+0000","last_peered":"2024-04-26T02:30:22.223882+0000","last_clean":"2024-04-26T02:30:22.223882+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:22.223882+0000","last_undegraded":"2024-04-26T02:30:22.223882+0000","last_fullsized":"2024-04-26T02:30:22.223882+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:26.846292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33698316,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7807","reported_seq":8022,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.234807+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:22.234807+0000","last_peered":"2024-04-26T02:30:22.234807+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:22.234807+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34255372,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7623","reported_seq":7835,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.323333+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:24.323333+0000","last_peered":"2024-04-26T02:30:24.323333+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:24.323333+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33719320,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7496","reported_seq":7637,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.594887+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:19.594887+0000","last_peered":"2024-04-26T02:30:19.594887+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:19.594887+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321484,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2011,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7618","reported_seq":7807,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.330746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:24.330746+0000","last_peered":"2024-04-26T02:30:24.330746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:24.330746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33590034,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7531","reported_seq":7729,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.397495+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:24.397495+0000","last_peered":"2024-04-26T02:30:24.397495+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:24.397495+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33306886,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7530,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2156}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7646","reported_seq":7836,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:19.337774+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:19.337774+0000","last_peered":"2024-04-26T02:30:19.337774+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:19.337774+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33855750,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7832","reported_seq":8038,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.229949+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:24.229949+0000","last_peered":"2024-04-26T02:30:24.229949+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:24.229949+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35140108,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7687","reported_seq":11892,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:23.305008+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:23.305008+0000","last_peered":"2024-04-26T02:30:23.305008+0000","last_clean":"2024-04-26T02:30:23.305008+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:23.305008+0000","last_undegraded":"2024-04-26T02:30:23.305008+0000","last_fullsized":"2024-04-26T02:30:23.305008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:26.485045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34140684,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7662","reported_seq":7852,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:22.144112+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:22.144112+0000","last_peered":"2024-04-26T02:30:22.144112+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:22.144112+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7662,"log_dups_size":0,"ondisk_log_size":7662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33870348,"num_objects":4135,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7660,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539671744,"num_objects":65885,"num_object_clones":0,"num_object_copies":263540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121672,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6484,"num_bytes_recovered":45883392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064988672,"data_stored":1064988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121699,"ondisk_log_size":121699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691368,"kb_used_data":262624,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93680472,"statfs":{"total":96636764160,"available":95928803328,"internally_reserved":0,"allocated":268926976,"data_stored":268732621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,4,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699532,"kb_used_data":267140,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93672308,"statfs":{"total":96636764160,"available":95920443392,"internally_reserved":0,"allocated":273551360,"data_stored":273357005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550580,"kb_used_data":248236,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93821260,"statfs":{"total":96636764160,"available":96072970240,"internally_reserved":0,"allocated":254193664,"data_stored":253999309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701644,"kb_used_data":264644,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93670196,"statfs":{"total":96636764160,"available":95918280704,"internally_reserved":0,"allocated":270995456,"data_stored":270801101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270295040,"data_stored":270295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253743104,"data_stored":253743104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272535552,"data_stored":272535552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268414976,"data_stored":268414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:26.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:26.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7470", 'reported_seq': 7673, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.414199+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:24.414199+0000', 'last_peered': '2024-04-26T02:30:24.414199+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:24.414199+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7470, 'log_dups_size': 0, 'ondisk_log_size': 7470, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33210368, 'num_objects': 4054, 'num_object_clones': 0, 'num_object_copies': 16216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7470, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2115}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7664", 'reported_seq': 7851, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:22.144845+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:22.144845+0000', 'last_peered': '2024-04-26T02:30:22.144845+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:22.144845+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7664, 'log_dups_size': 0, 'ondisk_log_size': 7664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34140684, 'num_objects': 4168, 'num_object_clones': 0, 'num_object_copies': 16672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7662, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7501", 'reported_seq': 7661, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:19.476898+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:19.476898+0000', 'last_peered': '2024-04-26T02:30:19.476898+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:19.476898+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7501, 'log_dups_size': 0, 'ondisk_log_size': 7501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33823006, 'num_objects': 4130, 'num_object_clones': 0, 'num_object_copies': 16520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1987, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4130, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7500, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2144}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'7427", 'reported_seq': 10468, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:22.255325+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:30:22.255325+0000', 'last_peered': '2024-04-26T02:30:22.255325+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:30:22.255325+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7427, 'log_dups_size': 0, 'ondisk_log_size': 7427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33198604, 'num_objects': 4053, 'num_object_clones': 0, 'num_object_copies': 16212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 484, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7425, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1488, 'num_bytes_recovered': 12009472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7522", 'reported_seq': 7718, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.316078+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:24.316078+0000', 'last_peered': '2024-04-26T02:30:24.316078+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:24.316078+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7522, 'log_dups_size': 0, 'ondisk_log_size': 7522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33378828, 'num_objects': 4075, 'num_object_clones': 0, 'num_object_copies': 16300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1946, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7520, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2131}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7569", 'reported_seq': 7789, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.401311+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:24.401311+0000', 'last_peered': '2024-04-26T02:30:24.401311+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:24.401311+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7569, 'log_dups_size': 0, 'ondisk_log_size': 7569, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33021952, 'num_objects': 4031, 'num_object_clones': 0, 'num_object_copies': 16124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7569, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7807", 'reported_seq': 8022, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:22.234807+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:22.234807+0000', 'last_peered': '2024-04-26T02:30:22.234807+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:22.234807+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7807, 'log_dups_size': 0, 'ondisk_log_size': 7807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34255372, 'num_objects': 4182, 'num_object_clones': 0, 'num_object_copies': 16728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7805, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7623", 'reported_seq': 7835, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.323333+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:30:24.323333+0000', 'last_peered': '2024-04-26T02:30:24.323333+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:24.323333+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7623, 'log_dups_size': 0, 'ondisk_log_size': 7623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33719320, 'num_objects': 4117, 'num_object_clones': 0, 'num_object_copies': 16468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1932, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7620, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2188}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7496", 'reported_seq': 7637, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:19.594887+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:19.594887+0000', 'last_peered': '2024-04-26T02:30:19.594887+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:19.594887+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7496, 'log_dups_size': 0, 'ondisk_log_size': 7496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33321484, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2011, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7494, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2059}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7618", 'reported_seq': 7807, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.330746+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:24.330746+0000', 'last_peered': '2024-04-26T02:30:24.330746+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:24.330746+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7618, 'log_dups_size': 0, 'ondisk_log_size': 7618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33590034, 'num_objects': 4101, 'num_object_clones': 0, 'num_object_copies': 16404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7615, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 66, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2147}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:26.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7531", 'reported_seq': 7729, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.397495+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:24.397495+0000', 'last_peered': '2024-04-26T02:30:24.397495+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:24.397495+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7531, 'log_dups_size': 0, 'ondisk_log_size': 7531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33306886, 'num_objects': 4066, 'num_object_clones': 0, 'num_object_copies': 16264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1911, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7530, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7646", 'reported_seq': 7836, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:19.337774+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:19.337774+0000', 'last_peered': '2024-04-26T02:30:19.337774+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:19.337774+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7646, 'log_dups_size': 0, 'ondisk_log_size': 7646, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33855750, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 16532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2057, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7832", 'reported_seq': 8038, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:24.229949+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:24.229949+0000', 'last_peered': '2024-04-26T02:30:24.229949+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:24.229949+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7832, 'log_dups_size': 0, 'ondisk_log_size': 7832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35140108, 'num_objects': 4290, 'num_object_clones': 0, 'num_object_copies': 17160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1999, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7830, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:26.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7662", 'reported_seq': 7852, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:22.144112+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:22.144112+0000', 'last_peered': '2024-04-26T02:30:22.144112+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:22.144112+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7662, 'log_dups_size': 0, 'ondisk_log_size': 7662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33870348, 'num_objects': 4135, 'num_object_clones': 0, 'num_object_copies': 16540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7660, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:26.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:28.443 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:28.443 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26279/262772 objects degraded (10.001%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":424,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65693,"data_bytes":538098880,"bytes_used":2710814720,"bytes_avail":383836241920,"bytes_total":386547056640,"degraded_objects":26279,"degraded_total":262772,"degraded_ratio":0.10000685004490585,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":139879,"recovering_keys_per_sec":0,"num_objects_recovered":300,"num_bytes_recovered":1679360,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-26T02:30:20.597425+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:28.526 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:28.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:30.028 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:30.153 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-04-26T02:30:28.602925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534609088,"num_objects":65267,"num_object_clones":0,"num_object_copies":261068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122290,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122317,"ondisk_log_size":122317,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2642084,"kb_used_data":1033284,"kb_used_omap":14,"kb_used_meta":1608753,"kb_avail":374845276,"statfs":{"total":386547056640,"available":383841562624,"internally_reserved":0,"allocated":1058082816,"data_stored":1057305396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1647363635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,9,4,10,5,0,0,7,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21585920,"num_objects":-2635,"num_object_clones":0,"num_object_copies":-10540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2635,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006692"},"pg_stats":[{"pgid":"1.f","version":"36'7470","reported_seq":7673,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.414199+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:24.414199+0000","last_peered":"2024-04-26T02:30:24.414199+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:24.414199+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7733","reported_seq":7924,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:27.366150+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:27.366150+0000","last_peered":"2024-04-26T02:30:27.366150+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:27.366150+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33575436,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7570","reported_seq":7730,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.578707+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:24.578707+0000","last_peered":"2024-04-26T02:30:24.578707+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:24.578707+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33257758,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7569,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7496","reported_seq":10829,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:27.314638+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:27.314638+0000","last_peered":"2024-04-26T02:30:27.314638+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:27.314638+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32633356,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7522","reported_seq":7718,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.316078+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:24.316078+0000","last_peered":"2024-04-26T02:30:24.316078+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:24.316078+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33378828,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7569","reported_seq":7789,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.401311+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:24.401311+0000","last_peered":"2024-04-26T02:30:24.401311+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:24.401311+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7719","reported_seq":11966,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:27.390022+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:27.390022+0000","last_peered":"2024-04-26T02:30:27.390022+0000","last_clean":"2024-04-26T02:30:27.390022+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:27.390022+0000","last_undegraded":"2024-04-26T02:30:27.390022+0000","last_fullsized":"2024-04-26T02:30:27.390022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083916,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7880","reported_seq":8097,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:27.312962+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:27.312962+0000","last_peered":"2024-04-26T02:30:27.312962+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:27.312962+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33657356,"num_objects":4109,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7623","reported_seq":7835,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.323333+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:24.323333+0000","last_peered":"2024-04-26T02:30:24.323333+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:24.323333+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33719320,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7550","reported_seq":7697,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.594980+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:24.594980+0000","last_peered":"2024-04-26T02:30:24.594980+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:24.594980+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32879116,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2008}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7618","reported_seq":7807,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.330746+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:24.330746+0000","last_peered":"2024-04-26T02:30:24.330746+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:24.330746+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33590034,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7531","reported_seq":7729,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.397495+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:24.397495+0000","last_peered":"2024-04-26T02:30:24.397495+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:24.397495+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33306886,"num_objects":4066,"num_object_clones":0,"num_object_copies":16264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7530,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2156}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7715","reported_seq":7910,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.418304+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:24.418304+0000","last_peered":"2024-04-26T02:30:24.418304+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:24.418304+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33290502,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7714,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2011}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7832","reported_seq":8038,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:24.229949+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:24.229949+0000","last_peered":"2024-04-26T02:30:24.229949+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:24.229949+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35140108,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7757","reported_seq":11962,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:28.227860+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:28.227860+0000","last_peered":"2024-04-26T02:30:28.227860+0000","last_clean":"2024-04-26T02:30:28.227860+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:28.227860+0000","last_undegraded":"2024-04-26T02:30:28.227860+0000","last_fullsized":"2024-04-26T02:30:28.227860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:31.554538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33567244,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7732","reported_seq":7922,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:27.165068+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:27.165068+0000","last_peered":"2024-04-26T02:30:27.165068+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:27.165068+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7732,"log_dups_size":0,"ondisk_log_size":7732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33296908,"num_objects":4065,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534609088,"num_objects":65267,"num_object_clones":0,"num_object_copies":261068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122290,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1055141888,"data_stored":1055141888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122317,"ondisk_log_size":122317,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691368,"kb_used_data":262624,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93680472,"statfs":{"total":96636764160,"available":95928803328,"internally_reserved":0,"allocated":268926976,"data_stored":268732621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,4,0,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703688,"kb_used_data":262976,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93668152,"statfs":{"total":96636764160,"available":95916187648,"internally_reserved":0,"allocated":269287424,"data_stored":269093069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,1,3,1,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558044,"kb_used_data":247380,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93813796,"statfs":{"total":96636764160,"available":96065327104,"internally_reserved":0,"allocated":253317120,"data_stored":253122765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688984,"kb_used_data":260304,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93682856,"statfs":{"total":96636764160,"available":95931244544,"internally_reserved":0,"allocated":266551296,"data_stored":266356941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,1,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265715712,"data_stored":265715712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252858368,"data_stored":252858368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":268152832,"data_stored":268152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268414976,"data_stored":268414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:33.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:34.471 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:34.471 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:34.573 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-04-26T02:30:32.603795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":526113984,"num_objects":64230,"num_object_clones":0,"num_object_copies":256920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123327,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6816,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123354,"ondisk_log_size":123354,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628516,"kb_used_data":1019696,"kb_used_omap":14,"kb_used_meta":1608753,"kb_avail":374858844,"statfs":{"total":386547056640,"available":383855456256,"internally_reserved":0,"allocated":1044168704,"data_stored":1043391284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1647363635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,14,6,10,3,2,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20897792,"num_objects":-2551,"num_object_clones":0,"num_object_copies":-10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006469"},"pg_stats":[{"pgid":"1.f","version":"36'7549","reported_seq":7756,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.331064+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:29.331064+0000","last_peered":"2024-04-26T02:30:29.331064+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:29.331064+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7807","reported_seq":8010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.210430+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:32.210430+0000","last_peered":"2024-04-26T02:30:32.210430+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:32.210430+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32969228,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7636","reported_seq":7798,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.669726+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:29.669726+0000","last_peered":"2024-04-26T02:30:29.669726+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:29.669726+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32717086,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7635,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7550","reported_seq":11150,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.409958+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:32.409958+0000","last_peered":"2024-04-26T02:30:32.409958+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:32.409958+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190988,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3725}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7593","reported_seq":7791,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.342783+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:29.342783+0000","last_peered":"2024-04-26T02:30:29.342783+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:29.342783+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32797196,"num_objects":4004,"num_object_clones":0,"num_object_copies":16016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7591,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7646","reported_seq":7875,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.336179+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:29.336179+0000","last_peered":"2024-04-26T02:30:29.336179+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:29.336179+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32391168,"num_objects":3954,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7802","reported_seq":12050,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:32.359110+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:32.359110+0000","last_peered":"2024-04-26T02:30:32.359110+0000","last_clean":"2024-04-26T02:30:32.359110+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:32.359110+0000","last_undegraded":"2024-04-26T02:30:32.359110+0000","last_fullsized":"2024-04-26T02:30:32.359110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7802,"log_dups_size":0,"ondisk_log_size":7802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:32.826295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32403980,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7950","reported_seq":8180,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.395737+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:32.395737+0000","last_peered":"2024-04-26T02:30:32.395737+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:32.395737+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7950,"log_dups_size":0,"ondisk_log_size":7950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083916,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7948,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1977}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7678","reported_seq":7896,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.266021+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:29.266021+0000","last_peered":"2024-04-26T02:30:29.266021+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:29.266021+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33268760,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7623","reported_seq":7773,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.595771+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:29.595771+0000","last_peered":"2024-04-26T02:30:29.595771+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:29.595771+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32281100,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7687","reported_seq":7882,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.358160+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:29.358160+0000","last_peered":"2024-04-26T02:30:29.358160+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:29.358160+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33024786,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7684,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7593","reported_seq":7795,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.294133+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:29.294133+0000","last_peered":"2024-04-26T02:30:29.294133+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:29.294133+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32798982,"num_objects":4004,"num_object_clones":0,"num_object_copies":16016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7592,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7788","reported_seq":7986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.464717+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:29.464717+0000","last_peered":"2024-04-26T02:30:29.464717+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:29.464717+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692486,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7787,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1939}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7899","reported_seq":8115,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.356842+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:29.356842+0000","last_peered":"2024-04-26T02:30:29.356842+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:29.356842+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7899,"log_dups_size":0,"ondisk_log_size":7899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34591244,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7757","reported_seq":11962,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:28.227860+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:28.227860+0000","last_peered":"2024-04-26T02:30:28.227860+0000","last_clean":"2024-04-26T02:30:28.227860+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:28.227860+0000","last_undegraded":"2024-04-26T02:30:28.227860+0000","last_fullsized":"2024-04-26T02:30:28.227860+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:31.554538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33567244,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7796","reported_seq":8004,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.354777+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:32.354777+0000","last_peered":"2024-04-26T02:30:32.354777+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:32.354777+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772620,"num_objects":4001,"num_object_clones":0,"num_object_copies":16004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":526113984,"num_objects":64230,"num_object_clones":0,"num_object_copies":256920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123327,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6816,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041367040,"data_stored":1041367040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123354,"ondisk_log_size":123354,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695060,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93676780,"statfs":{"total":96636764160,"available":95925022720,"internally_reserved":0,"allocated":264183808,"data_stored":263989453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686640,"kb_used_data":258392,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93685200,"statfs":{"total":96636764160,"available":95933644800,"internally_reserved":0,"allocated":264593408,"data_stored":264399053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558044,"kb_used_data":247380,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93813796,"statfs":{"total":96636764160,"available":96065327104,"internally_reserved":0,"allocated":253317120,"data_stored":253122765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688772,"kb_used_data":255932,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93683068,"statfs":{"total":96636764160,"available":95931461632,"internally_reserved":0,"allocated":262074368,"data_stored":261880013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261222400,"data_stored":261222400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252858368,"data_stored":252858368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263524352,"data_stored":263524352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":263761920,"data_stored":263761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:34.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:36.060 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:36.060 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:36.121 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-04-26T02:30:34.605204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525565120,"num_objects":64163,"num_object_clones":0,"num_object_copies":256652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123394,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6816,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123421,"ondisk_log_size":123421,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2633224,"kb_used_data":1018836,"kb_used_omap":14,"kb_used_meta":1614321,"kb_avail":374854136,"statfs":{"total":386547056640,"available":383850635264,"internally_reserved":0,"allocated":1043288064,"data_stored":1042510644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1653065267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,15,7,12,3,2,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18571264,"num_objects":-2267,"num_object_clones":0,"num_object_copies":-9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2267,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007340"},"pg_stats":[{"pgid":"1.f","version":"36'7549","reported_seq":7756,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.331064+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:29.331064+0000","last_peered":"2024-04-26T02:30:29.331064+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:29.331064+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7807","reported_seq":8010,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.210430+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:32.210430+0000","last_peered":"2024-04-26T02:30:32.210430+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:32.210430+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32969228,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7636","reported_seq":7798,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.669726+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:29.669726+0000","last_peered":"2024-04-26T02:30:29.669726+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:29.669726+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32717086,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7635,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7550","reported_seq":11150,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.409958+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:32.409958+0000","last_peered":"2024-04-26T02:30:32.409958+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:32.409958+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190988,"num_objects":3930,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3725}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7593","reported_seq":7791,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.342783+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:29.342783+0000","last_peered":"2024-04-26T02:30:29.342783+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:29.342783+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32797196,"num_objects":4004,"num_object_clones":0,"num_object_copies":16016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7591,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2061}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7646","reported_seq":7875,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.336179+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:29.336179+0000","last_peered":"2024-04-26T02:30:29.336179+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:29.336179+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32391168,"num_objects":3954,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7802","reported_seq":12050,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:32.359110+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:32.359110+0000","last_peered":"2024-04-26T02:30:32.359110+0000","last_clean":"2024-04-26T02:30:32.359110+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:32.359110+0000","last_undegraded":"2024-04-26T02:30:32.359110+0000","last_fullsized":"2024-04-26T02:30:32.359110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7802,"log_dups_size":0,"ondisk_log_size":7802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:32.826295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32403980,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'7950","reported_seq":8180,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.395737+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:32.395737+0000","last_peered":"2024-04-26T02:30:32.395737+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:32.395737+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7950,"log_dups_size":0,"ondisk_log_size":7950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083916,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7948,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1977}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7678","reported_seq":7896,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.266021+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:29.266021+0000","last_peered":"2024-04-26T02:30:29.266021+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:29.266021+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33268760,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7623","reported_seq":7773,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.595771+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:29.595771+0000","last_peered":"2024-04-26T02:30:29.595771+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:29.595771+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32281100,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7687","reported_seq":7882,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.358160+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:29.358160+0000","last_peered":"2024-04-26T02:30:29.358160+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:29.358160+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33024786,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7684,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7593","reported_seq":7795,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.294133+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:29.294133+0000","last_peered":"2024-04-26T02:30:29.294133+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:29.294133+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32798982,"num_objects":4004,"num_object_clones":0,"num_object_copies":16016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7592,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7788","reported_seq":7986,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.464717+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:29.464717+0000","last_peered":"2024-04-26T02:30:29.464717+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:29.464717+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692486,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7787,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1939}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7899","reported_seq":8115,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:29.356842+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:29.356842+0000","last_peered":"2024-04-26T02:30:29.356842+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:29.356842+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7899,"log_dups_size":0,"ondisk_log_size":7899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34591244,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7824","reported_seq":12029,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:33.229585+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:33.229585+0000","last_peered":"2024-04-26T02:30:33.229585+0000","last_clean":"2024-04-26T02:30:33.229585+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:33.229585+0000","last_undegraded":"2024-04-26T02:30:33.229585+0000","last_fullsized":"2024-04-26T02:30:33.229585+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:36.614166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33018380,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7796","reported_seq":8004,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:32.354777+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:32.354777+0000","last_peered":"2024-04-26T02:30:32.354777+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:32.354777+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772620,"num_objects":4001,"num_object_clones":0,"num_object_copies":16004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":525565120,"num_objects":64163,"num_object_clones":0,"num_object_copies":256652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123394,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6816,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040510976,"data_stored":1040510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123421,"ondisk_log_size":123421,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695060,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93676780,"statfs":{"total":96636764160,"available":95925022720,"internally_reserved":0,"allocated":264183808,"data_stored":263989453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686640,"kb_used_data":258392,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93685200,"statfs":{"total":96636764160,"available":95933644800,"internally_reserved":0,"allocated":264593408,"data_stored":264399053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562752,"kb_used_data":246520,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93809088,"statfs":{"total":96636764160,"available":96060506112,"internally_reserved":0,"allocated":252436480,"data_stored":252242125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323808975},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688772,"kb_used_data":255932,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93683068,"statfs":{"total":96636764160,"available":95931461632,"internally_reserved":0,"allocated":262074368,"data_stored":261880013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,1,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":261222400,"data_stored":261222400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252002304,"data_stored":252002304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263524352,"data_stored":263524352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":263761920,"data_stored":263761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:36.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:36.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7549", 'reported_seq': 7756, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.331064+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:29.331064+0000', 'last_peered': '2024-04-26T02:30:29.331064+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:29.331064+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7549, 'log_dups_size': 0, 'ondisk_log_size': 7549, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32563200, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7549, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:36.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "36'7807", 'reported_seq': 8010, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:32.210430+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:32.210430+0000', 'last_peered': '2024-04-26T02:30:32.210430+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:32.210430+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7807, 'log_dups_size': 0, 'ondisk_log_size': 7807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32969228, 'num_objects': 4025, 'num_object_clones': 0, 'num_object_copies': 16100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2012, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7805, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:36.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7636", 'reported_seq': 7798, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.669726+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:29.669726+0000', 'last_peered': '2024-04-26T02:30:29.669726+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:29.669726+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7636, 'log_dups_size': 0, 'ondisk_log_size': 7636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32717086, 'num_objects': 3995, 'num_object_clones': 0, 'num_object_copies': 15980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3995, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7635, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2010}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "36'7550", 'reported_seq': 11150, 'reported_epoch': 36, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:32.409958+0000', 'last_change': '2024-04-26T02:28:51.531328+0000', 'last_active': '2024-04-26T02:30:32.409958+0000', 'last_peered': '2024-04-26T02:30:32.409958+0000', 'last_clean': '2024-04-26T02:23:50.847702+0000', 'last_became_active': '2024-04-26T02:24:59.556130+0000', 'last_became_peered': '2024-04-26T02:24:59.556130+0000', 'last_unstale': '2024-04-26T02:30:32.409958+0000', 'last_undegraded': '2024-04-26T02:24:59.274885+0000', 'last_fullsized': '2024-04-26T02:24:59.231813+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7550, 'log_dups_size': 0, 'ondisk_log_size': 7550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32190988, 'num_objects': 3930, 'num_object_clones': 0, 'num_object_copies': 15720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7548, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1767, 'num_bytes_recovered': 12034048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3725}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7593", 'reported_seq': 7791, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.342783+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:29.342783+0000', 'last_peered': '2024-04-26T02:30:29.342783+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:29.342783+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7593, 'log_dups_size': 0, 'ondisk_log_size': 7593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32797196, 'num_objects': 4004, 'num_object_clones': 0, 'num_object_copies': 16016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7591, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 70, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2061}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7646", 'reported_seq': 7875, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.336179+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:29.336179+0000', 'last_peered': '2024-04-26T02:30:29.336179+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:29.336179+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7646, 'log_dups_size': 0, 'ondisk_log_size': 7646, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32391168, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 15816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1961, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7646, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1993}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:36.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "36'7950", 'reported_seq': 8180, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:32.395737+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:32.395737+0000', 'last_peered': '2024-04-26T02:30:32.395737+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:32.395737+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7950, 'log_dups_size': 0, 'ondisk_log_size': 7950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33083916, 'num_objects': 4039, 'num_object_clones': 0, 'num_object_copies': 16156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7948, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7678", 'reported_seq': 7896, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.266021+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:30:29.266021+0000', 'last_peered': '2024-04-26T02:30:29.266021+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:29.266021+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7678, 'log_dups_size': 0, 'ondisk_log_size': 7678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33268760, 'num_objects': 4062, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1929, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7675, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2136}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7623", 'reported_seq': 7773, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.595771+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:29.595771+0000', 'last_peered': '2024-04-26T02:30:29.595771+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:29.595771+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7623, 'log_dups_size': 0, 'ondisk_log_size': 7623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32281100, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7621, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7687", 'reported_seq': 7882, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.358160+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:29.358160+0000', 'last_peered': '2024-04-26T02:30:29.358160+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:29.358160+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7687, 'log_dups_size': 0, 'ondisk_log_size': 7687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33024786, 'num_objects': 4032, 'num_object_clones': 0, 'num_object_copies': 16128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1954, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7684, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2081}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7593", 'reported_seq': 7795, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.294133+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:29.294133+0000', 'last_peered': '2024-04-26T02:30:29.294133+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:29.294133+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7593, 'log_dups_size': 0, 'ondisk_log_size': 7593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32798982, 'num_objects': 4004, 'num_object_clones': 0, 'num_object_copies': 16016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1910, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7592, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2095}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:36.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7788", 'reported_seq': 7986, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.464717+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:29.464717+0000', 'last_peered': '2024-04-26T02:30:29.464717+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:29.464717+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32692486, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 15964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7787, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:36.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:36.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'7899", 'reported_seq': 8115, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:29.356842+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:29.356842+0000', 'last_peered': '2024-04-26T02:30:29.356842+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:29.356842+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7899, 'log_dups_size': 0, 'ondisk_log_size': 7899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34591244, 'num_objects': 4223, 'num_object_clones': 0, 'num_object_copies': 16892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1994, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7897, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:36.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:36.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "36'7796", 'reported_seq': 8004, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:32.354777+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:32.354777+0000', 'last_peered': '2024-04-26T02:30:32.354777+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:32.354777+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7796, 'log_dups_size': 0, 'ondisk_log_size': 7796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32772620, 'num_objects': 4001, 'num_object_clones': 0, 'num_object_copies': 16004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7794, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:36.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:37.172 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25897/253948 objects degraded (10.198%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63487,"data_bytes":520027328,"bytes_used":2678927360,"bytes_avail":383868129280,"bytes_total":386547056640,"degraded_objects":25897,"degraded_total":253948,"degraded_ratio":0.10197757021122435,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":2046,"recovering_keys_per_sec":0,"num_objects_recovered":388,"num_bytes_recovered":24576,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:37.298 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:37.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:38.884 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:38.884 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:38.964 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-04-26T02:30:38.606253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516652224,"num_objects":63075,"num_object_clones":0,"num_object_copies":252300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124482,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7012,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124509,"ondisk_log_size":124509,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2641740,"kb_used_data":1005164,"kb_used_omap":14,"kb_used_meta":1636529,"kb_avail":374845620,"statfs":{"total":386547056640,"available":383841914880,"internally_reserved":0,"allocated":1029287936,"data_stored":1028510516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1675806259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,5,9,11,8,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21446656,"num_objects":-2618,"num_object_clones":0,"num_object_copies":-10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":522,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007352"},"pg_stats":[{"pgid":"1.f","version":"36'7614","reported_seq":7833,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.476415+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:34.476415+0000","last_peered":"2024-04-26T02:30:34.476415+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:34.476415+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7614,"log_dups_size":0,"ondisk_log_size":7614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32030720,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'7890","reported_seq":8103,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:37.377352+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:37.377352+0000","last_peered":"2024-04-26T02:30:37.377352+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:37.377352+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32289292,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7702","reported_seq":7867,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.736414+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:34.736414+0000","last_peered":"2024-04-26T02:30:34.736414+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:34.736414+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7702,"log_dups_size":0,"ondisk_log_size":7702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32176414,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7701,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7629","reported_seq":11492,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:37.424731+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:37.424731+0000","last_peered":"2024-04-26T02:30:37.424731+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:37.424731+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31543820,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1899,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7661","reported_seq":7865,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.428607+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:34.428607+0000","last_peered":"2024-04-26T02:30:34.428607+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:34.428607+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7661,"log_dups_size":0,"ondisk_log_size":7661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32240140,"num_objects":3936,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7659,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1996}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7720","reported_seq":7955,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.478566+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:34.478566+0000","last_peered":"2024-04-26T02:30:34.478566+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:34.478566+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7720,"log_dups_size":0,"ondisk_log_size":7720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"36'7860","reported_seq":12108,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:37.272464+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:37.272464+0000","last_peered":"2024-04-26T02:30:37.272464+0000","last_clean":"2024-04-26T02:30:37.272464+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:37.272464+0000","last_undegraded":"2024-04-26T02:30:37.272464+0000","last_fullsized":"2024-04-26T02:30:37.272464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7860,"log_dups_size":0,"ondisk_log_size":7860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:38.806566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31928844,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'8018","reported_seq":8252,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:37.401089+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:37.401089+0000","last_peered":"2024-04-26T02:30:37.401089+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:37.401089+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32526860,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8016,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7747","reported_seq":7992,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.522140+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:34.522140+0000","last_peered":"2024-04-26T02:30:34.522140+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:34.522140+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32703512,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1916,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7687","reported_seq":7845,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.677239+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:34.677239+0000","last_peered":"2024-04-26T02:30:34.677239+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:34.677239+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31756812,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7743","reported_seq":7951,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.556518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:34.556518+0000","last_peered":"2024-04-26T02:30:34.556518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:34.556518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7743,"log_dups_size":0,"ondisk_log_size":7743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32566034,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7740,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7664","reported_seq":7882,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.610387+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:34.610387+0000","last_peered":"2024-04-26T02:30:34.610387+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:34.610387+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7664,"log_dups_size":0,"ondisk_log_size":7664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32217350,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7663,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2032}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7863","reported_seq":8068,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.751849+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:34.751849+0000","last_peered":"2024-04-26T02:30:34.751849+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:34.751849+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7863,"log_dups_size":0,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078086,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1867}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'7967","reported_seq":8201,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:34.512149+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:34.512149+0000","last_peered":"2024-04-26T02:30:34.512149+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:34.512149+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34034188,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2172}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7889","reported_seq":12093,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:38.312650+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:38.312650+0000","last_peered":"2024-04-26T02:30:38.312650+0000","last_clean":"2024-04-26T02:30:38.312650+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:38.312650+0000","last_undegraded":"2024-04-26T02:30:38.312650+0000","last_fullsized":"2024-04-26T02:30:38.312650+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:42.625251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32485900,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"36'7855","reported_seq":8065,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:37.400841+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:37.400841+0000","last_peered":"2024-04-26T02:30:37.400841+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:37.400841+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7855,"log_dups_size":0,"ondisk_log_size":7855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32289292,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7853,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516652224,"num_objects":63075,"num_object_clones":0,"num_object_copies":252300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124482,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7012,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1026424832,"data_stored":1026424832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124509,"ondisk_log_size":124509,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117072,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678180,"kb_used_data":253596,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93693660,"statfs":{"total":96636764160,"available":95942307840,"internally_reserved":0,"allocated":259682304,"data_stored":259487949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,5,4,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686436,"kb_used_data":254044,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93685404,"statfs":{"total":96636764160,"available":95933853696,"internally_reserved":0,"allocated":260141056,"data_stored":259946701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570300,"kb_used_data":245748,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93801540,"statfs":{"total":96636764160,"available":96052776960,"internally_reserved":0,"allocated":251645952,"data_stored":251451597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706824,"kb_used_data":251776,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93665016,"statfs":{"total":96636764160,"available":95912976384,"internally_reserved":0,"allocated":257818624,"data_stored":257624269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":256827392,"data_stored":256827392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251224064,"data_stored":251224064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259125248,"data_stored":259125248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259248128,"data_stored":259248128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:41.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:43.275 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:43.381 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-04-26T02:30:42.607294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508746944,"num_objects":62110,"num_object_clones":0,"num_object_copies":248440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125447,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7052,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125474,"ondisk_log_size":125474,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673060,"kb_used_data":992064,"kb_used_omap":14,"kb_used_meta":1680945,"kb_avail":374814300,"statfs":{"total":386547056640,"available":383809843200,"internally_reserved":0,"allocated":1015873536,"data_stored":1015091473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1721288243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,8,1,1,8,2,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16818176,"num_objects":-2053,"num_object_clones":0,"num_object_copies":-8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002089"},"pg_stats":[{"pgid":"1.f","version":"36'7695","reported_seq":7915,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.714733+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:39.714733+0000","last_peered":"2024-04-26T02:30:39.714733+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:39.714733+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31367168,"num_objects":3829,"num_object_clones":0,"num_object_copies":15316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7959","reported_seq":8184,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.513274+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:42.513274+0000","last_peered":"2024-04-26T02:30:42.513274+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:42.513274+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724044,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7773","reported_seq":7938,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.517404+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:39.517404+0000","last_peered":"2024-04-26T02:30:39.517404+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:39.517404+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7773,"log_dups_size":0,"ondisk_log_size":7773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31594782,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7772,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"36'7629","reported_seq":11492,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:37.424731+0000","last_change":"2024-04-26T02:28:51.531328+0000","last_active":"2024-04-26T02:30:37.424731+0000","last_peered":"2024-04-26T02:30:37.424731+0000","last_clean":"2024-04-26T02:23:50.847702+0000","last_became_active":"2024-04-26T02:24:59.556130+0000","last_became_peered":"2024-04-26T02:24:59.556130+0000","last_unstale":"2024-04-26T02:30:37.424731+0000","last_undegraded":"2024-04-26T02:24:59.274885+0000","last_fullsized":"2024-04-26T02:24:59.231813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31543820,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1899,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3778}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"36'7738","reported_seq":7948,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.617680+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:39.617680+0000","last_peered":"2024-04-26T02:30:39.617680+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:39.617680+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609356,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7777","reported_seq":8022,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.613407+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:39.613407+0000","last_peered":"2024-04-26T02:30:39.613407+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:39.613407+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31318016,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7777,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7923","reported_seq":12175,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:42.200871+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:42.200871+0000","last_peered":"2024-04-26T02:30:42.200871+0000","last_clean":"2024-04-26T02:30:42.200871+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:42.200871+0000","last_undegraded":"2024-04-26T02:30:42.200871+0000","last_fullsized":"2024-04-26T02:30:42.200871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:44.805792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31412748,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8094","reported_seq":8344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.394837+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:42.394837+0000","last_peered":"2024-04-26T02:30:42.394837+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:42.394837+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31904268,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7815","reported_seq":8066,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.517475+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:39.517475+0000","last_peered":"2024-04-26T02:30:39.517475+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:39.517475+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32146456,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7746","reported_seq":7913,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.746349+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:39.746349+0000","last_peered":"2024-04-26T02:30:39.746349+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:39.746349+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7746,"log_dups_size":0,"ondisk_log_size":7746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31273484,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1822}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7811","reported_seq":8029,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.609013+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:39.609013+0000","last_peered":"2024-04-26T02:30:39.609013+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:39.609013+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32008978,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7723","reported_seq":7945,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.601175+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:39.601175+0000","last_peered":"2024-04-26T02:30:39.601175+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:39.601175+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31734022,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7918","reported_seq":8125,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.565811+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:39.565811+0000","last_peered":"2024-04-26T02:30:39.565811+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:39.565811+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7918,"log_dups_size":0,"ondisk_log_size":7918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31627526,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'8051","reported_seq":8285,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.690416+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:39.690416+0000","last_peered":"2024-04-26T02:30:39.690416+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:39.690416+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8051,"log_dups_size":0,"ondisk_log_size":8051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33346060,"num_objects":4071,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8049,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'7889","reported_seq":12093,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-26T02:30:38.312650+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:38.312650+0000","last_peered":"2024-04-26T02:30:38.312650+0000","last_clean":"2024-04-26T02:30:38.312650+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:38.312650+0000","last_undegraded":"2024-04-26T02:30:38.312650+0000","last_fullsized":"2024-04-26T02:30:38.312650+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:42.625251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32485900,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'7933","reported_seq":8159,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.415343+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:42.415343+0000","last_peered":"2024-04-26T02:30:42.415343+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:42.415343+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31650316,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":508746944,"num_objects":62110,"num_object_clones":0,"num_object_copies":248440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125447,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7052,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1013272576,"data_stored":1013272576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125474,"ondisk_log_size":125474,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700152,"kb_used_data":249200,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93671688,"statfs":{"total":96636764160,"available":95919808512,"internally_reserved":0,"allocated":255180800,"data_stored":254986445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":461762256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,8,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708476,"kb_used_data":249716,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":93663364,"statfs":{"total":96636764160,"available":95911284736,"internally_reserved":0,"allocated":255709184,"data_stored":255514829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570300,"kb_used_data":245748,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93801540,"statfs":{"total":96636764160,"available":96052776960,"internally_reserved":0,"allocated":251645952,"data_stored":251451597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694132,"kb_used_data":247400,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93677708,"statfs":{"total":96636764160,"available":95925972992,"internally_reserved":0,"allocated":253337600,"data_stored":253138602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252452864,"data_stored":252452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251224064,"data_stored":251224064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254763008,"data_stored":254763008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":254832640,"data_stored":254832640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:43.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:44.807 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:44.807 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:44.898 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-04-26T02:30:44.610013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507641024,"num_objects":61975,"num_object_clones":0,"num_object_copies":247900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125582,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7125,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125609,"ondisk_log_size":125609,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676624,"kb_used_data":991468,"kb_used_omap":14,"kb_used_meta":1685105,"kb_avail":374810736,"statfs":{"total":386547056640,"available":383806193664,"internally_reserved":0,"allocated":1015263232,"data_stored":1014471862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1725548083},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,8,1,1,8,2,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12386304,"num_objects":-1512,"num_object_clones":0,"num_object_copies":-6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004283"},"pg_stats":[{"pgid":"1.f","version":"36'7695","reported_seq":7915,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.714733+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:39.714733+0000","last_peered":"2024-04-26T02:30:39.714733+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:39.714733+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31367168,"num_objects":3829,"num_object_clones":0,"num_object_copies":15316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7959","reported_seq":8184,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.513274+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:42.513274+0000","last_peered":"2024-04-26T02:30:42.513274+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:42.513274+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724044,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'7773","reported_seq":7938,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.517404+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:39.517404+0000","last_peered":"2024-04-26T02:30:39.517404+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:39.517404+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7773,"log_dups_size":0,"ondisk_log_size":7773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31594782,"num_objects":3858,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7772,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7694","reported_seq":11719,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:43.395660+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:43.395660+0000","last_peered":"2024-04-26T02:30:43.395660+0000","last_clean":"2024-04-26T02:30:43.395660+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:43.395660+0000","last_undegraded":"2024-04-26T02:30:43.395660+0000","last_fullsized":"2024-04-26T02:30:43.395660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:47.540958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31011340,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"36'7738","reported_seq":7948,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.617680+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:39.617680+0000","last_peered":"2024-04-26T02:30:39.617680+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:39.617680+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609356,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'7777","reported_seq":8022,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.613407+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:39.613407+0000","last_peered":"2024-04-26T02:30:39.613407+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:39.613407+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31318016,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7777,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7923","reported_seq":12175,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:42.200871+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:42.200871+0000","last_peered":"2024-04-26T02:30:42.200871+0000","last_clean":"2024-04-26T02:30:42.200871+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:42.200871+0000","last_undegraded":"2024-04-26T02:30:42.200871+0000","last_fullsized":"2024-04-26T02:30:42.200871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:44.805792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31412748,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8094","reported_seq":8344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.394837+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:42.394837+0000","last_peered":"2024-04-26T02:30:42.394837+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:42.394837+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31904268,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'7815","reported_seq":8066,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.517475+0000","last_change":"2024-04-26T02:24:59.566874+0000","last_active":"2024-04-26T02:30:39.517475+0000","last_peered":"2024-04-26T02:30:39.517475+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:39.517475+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32146456,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'7746","reported_seq":7913,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.746349+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:39.746349+0000","last_peered":"2024-04-26T02:30:39.746349+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:39.746349+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7746,"log_dups_size":0,"ondisk_log_size":7746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31273484,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1822}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'7811","reported_seq":8029,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.609013+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:39.609013+0000","last_peered":"2024-04-26T02:30:39.609013+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:39.609013+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32008978,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"36'7723","reported_seq":7945,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.601175+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:39.601175+0000","last_peered":"2024-04-26T02:30:39.601175+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:39.601175+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31734022,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7722,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'7918","reported_seq":8125,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.565811+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:39.565811+0000","last_peered":"2024-04-26T02:30:39.565811+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:39.565811+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7918,"log_dups_size":0,"ondisk_log_size":7918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31627526,"num_objects":3861,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"36'8051","reported_seq":8285,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:39.690416+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:39.690416+0000","last_peered":"2024-04-26T02:30:39.690416+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:39.690416+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8051,"log_dups_size":0,"ondisk_log_size":8051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33346060,"num_objects":4071,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8049,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7959","reported_seq":12168,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:43.378881+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:43.378881+0000","last_peered":"2024-04-26T02:30:43.378881+0000","last_clean":"2024-04-26T02:30:43.378881+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:43.378881+0000","last_undegraded":"2024-04-26T02:30:43.378881+0000","last_fullsized":"2024-04-26T02:30:43.378881+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31912460,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'7933","reported_seq":8159,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.415343+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:42.415343+0000","last_peered":"2024-04-26T02:30:42.415343+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:42.415343+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31650316,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":507641024,"num_objects":61975,"num_object_clones":0,"num_object_copies":247900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125582,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7125,"num_bytes_recovered":45907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1012547584,"data_stored":1012547584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125609,"ondisk_log_size":125609,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700152,"kb_used_data":249200,"kb_used_omap":4,"kb_used_meta":450939,"kb_avail":93671688,"statfs":{"total":96636764160,"available":95919808512,"internally_reserved":0,"allocated":255180800,"data_stored":254986445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":461762256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,8,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708476,"kb_used_data":249716,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":93663364,"statfs":{"total":96636764160,"available":95911284736,"internally_reserved":0,"allocated":255709184,"data_stored":255514829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573864,"kb_used_data":245152,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93797976,"statfs":{"total":96636764160,"available":96049127424,"internally_reserved":0,"allocated":251035648,"data_stored":250831986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694132,"kb_used_data":247400,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93677708,"statfs":{"total":96636764160,"available":95925972992,"internally_reserved":0,"allocated":253337600,"data_stored":253138602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252452864,"data_stored":252452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250499072,"data_stored":250499072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254763008,"data_stored":254763008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":254832640,"data_stored":254832640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "36'7695", 'reported_seq': 7915, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.714733+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:39.714733+0000', 'last_peered': '2024-04-26T02:30:39.714733+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:39.714733+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7695, 'log_dups_size': 0, 'ondisk_log_size': 7695, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31367168, 'num_objects': 3829, 'num_object_clones': 0, 'num_object_copies': 15316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7695, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1899}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'7959", 'reported_seq': 8184, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:42.513274+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:42.513274+0000', 'last_peered': '2024-04-26T02:30:42.513274+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:42.513274+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7959, 'log_dups_size': 0, 'ondisk_log_size': 7959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31724044, 'num_objects': 3873, 'num_object_clones': 0, 'num_object_copies': 15492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2003, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7957, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "36'7773", 'reported_seq': 7938, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.517404+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:39.517404+0000', 'last_peered': '2024-04-26T02:30:39.517404+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:39.517404+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7773, 'log_dups_size': 0, 'ondisk_log_size': 7773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31594782, 'num_objects': 3858, 'num_object_clones': 0, 'num_object_copies': 15432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3858, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7772, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1874}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "36'7738", 'reported_seq': 7948, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.617680+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:39.617680+0000', 'last_peered': '2024-04-26T02:30:39.617680+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:39.617680+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7738, 'log_dups_size': 0, 'ondisk_log_size': 7738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31609356, 'num_objects': 3859, 'num_object_clones': 0, 'num_object_copies': 15436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7736, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1922}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "36'7777", 'reported_seq': 8022, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.613407+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:39.613407+0000', 'last_peered': '2024-04-26T02:30:39.613407+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:39.613407+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7777, 'log_dups_size': 0, 'ondisk_log_size': 7777, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31318016, 'num_objects': 3823, 'num_object_clones': 0, 'num_object_copies': 15292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7777, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1870}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8094", 'reported_seq': 8344, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:42.394837+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:42.394837+0000', 'last_peered': '2024-04-26T02:30:42.394837+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:42.394837+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8094, 'log_dups_size': 0, 'ondisk_log_size': 8094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31904268, 'num_objects': 3895, 'num_object_clones': 0, 'num_object_copies': 15580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8092, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 89, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "36'7815", 'reported_seq': 8066, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.517475+0000', 'last_change': '2024-04-26T02:24:59.566874+0000', 'last_active': '2024-04-26T02:30:39.517475+0000', 'last_peered': '2024-04-26T02:30:39.517475+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:39.517475+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7815, 'log_dups_size': 0, 'ondisk_log_size': 7815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32146456, 'num_objects': 3925, 'num_object_clones': 0, 'num_object_copies': 15700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1913, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7812, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2015}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "36'7746", 'reported_seq': 7913, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.746349+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:39.746349+0000', 'last_peered': '2024-04-26T02:30:39.746349+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:39.746349+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7746, 'log_dups_size': 0, 'ondisk_log_size': 7746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31273484, 'num_objects': 3818, 'num_object_clones': 0, 'num_object_copies': 15272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1998, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7744, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1822}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "36'7811", 'reported_seq': 8029, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.609013+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:39.609013+0000', 'last_peered': '2024-04-26T02:30:39.609013+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:39.609013+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7811, 'log_dups_size': 0, 'ondisk_log_size': 7811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32008978, 'num_objects': 3908, 'num_object_clones': 0, 'num_object_copies': 15632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1943, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7808, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "36'7723", 'reported_seq': 7945, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.601175+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:39.601175+0000', 'last_peered': '2024-04-26T02:30:39.601175+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:39.601175+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7723, 'log_dups_size': 0, 'ondisk_log_size': 7723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31734022, 'num_objects': 3874, 'num_object_clones': 0, 'num_object_copies': 15496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1900, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7722, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "36'7918", 'reported_seq': 8125, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.565811+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:39.565811+0000', 'last_peered': '2024-04-26T02:30:39.565811+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:39.565811+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7918, 'log_dups_size': 0, 'ondisk_log_size': 7918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31627526, 'num_objects': 3861, 'num_object_clones': 0, 'num_object_copies': 15444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7917, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 64, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:44.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:44.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "36'8051", 'reported_seq': 8285, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:39.690416+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:39.690416+0000', 'last_peered': '2024-04-26T02:30:39.690416+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:39.690416+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8051, 'log_dups_size': 0, 'ondisk_log_size': 8051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33346060, 'num_objects': 4071, 'num_object_clones': 0, 'num_object_copies': 16284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8049, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:44.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:44.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'7933", 'reported_seq': 8159, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:42.415343+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:42.415343+0000', 'last_peered': '2024-04-26T02:30:42.415343+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:42.415343+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7933, 'log_dups_size': 0, 'ondisk_log_size': 7933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31650316, 'num_objects': 3864, 'num_object_clones': 0, 'num_object_copies': 15456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7931, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:44.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:46.333 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:46.333 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25644/247900 objects degraded (10.344%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":442,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":3}],"num_pgs":16,"num_pools":1,"num_objects":61975,"data_bytes":507641024,"bytes_used":2740862976,"bytes_avail":383806193664,"bytes_total":386547056640,"degraded_objects":25644,"degraded_total":247900,"degraded_ratio":0.10344493747478822,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":253,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:46.368 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:46.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:47.796 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:47.797 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:48.005 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-04-26T02:30:46.610644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502234304,"num_objects":61315,"num_object_clones":0,"num_object_copies":245260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126242,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7220,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126269,"ondisk_log_size":126269,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659876,"kb_used_data":983028,"kb_used_omap":14,"kb_used_meta":1676785,"kb_avail":374827484,"statfs":{"total":386547056640,"available":383823343616,"internally_reserved":0,"allocated":1006620672,"data_stored":1005810688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1717028403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,6,4,10,17,0,0,0,1,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14417920,"num_objects":-1760,"num_object_clones":0,"num_object_copies":-7040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004391"},"pg_stats":[{"pgid":"1.f","version":"38'7770","reported_seq":8001,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.807911+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:44.807911+0000","last_peered":"2024-04-26T02:30:44.807911+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:44.807911+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7770,"log_dups_size":0,"ondisk_log_size":7770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30752768,"num_objects":3754,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7770,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'7959","reported_seq":8184,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.513274+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:42.513274+0000","last_peered":"2024-04-26T02:30:42.513274+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:42.513274+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31724044,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'7836","reported_seq":8005,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.601229+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:44.601229+0000","last_peered":"2024-04-26T02:30:44.601229+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:44.601229+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31078686,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7835,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1811}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7694","reported_seq":11719,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:43.395660+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:43.395660+0000","last_peered":"2024-04-26T02:30:43.395660+0000","last_clean":"2024-04-26T02:30:43.395660+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:43.395660+0000","last_undegraded":"2024-04-26T02:30:43.395660+0000","last_fullsized":"2024-04-26T02:30:43.395660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:47.540958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31011340,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7798","reported_seq":8022,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.642218+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:44.642218+0000","last_peered":"2024-04-26T02:30:44.642218+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:44.642218+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31117836,"num_objects":3799,"num_object_clones":0,"num_object_copies":15196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1867}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7844","reported_seq":8097,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.570364+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:44.570364+0000","last_peered":"2024-04-26T02:30:44.570364+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:44.570364+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7844,"log_dups_size":0,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30769152,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7923","reported_seq":12175,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:42.200871+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:42.200871+0000","last_peered":"2024-04-26T02:30:42.200871+0000","last_clean":"2024-04-26T02:30:42.200871+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:42.200871+0000","last_undegraded":"2024-04-26T02:30:42.200871+0000","last_fullsized":"2024-04-26T02:30:42.200871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:44.805792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31412748,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8094","reported_seq":8344,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.394837+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:42.394837+0000","last_peered":"2024-04-26T02:30:42.394837+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:42.394837+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31904268,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7874","reported_seq":8277,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.811501+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:30:44.811501+0000","last_peered":"2024-04-26T02:30:44.811501+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:44.811501+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31663128,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1839,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7814","reported_seq":7987,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.669676+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:44.669676+0000","last_peered":"2024-04-26T02:30:44.669676+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:44.669676+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7814,"log_dups_size":0,"ondisk_log_size":7814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30716428,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7873","reported_seq":8105,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.727429+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:44.727429+0000","last_peered":"2024-04-26T02:30:44.727429+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:44.727429+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31501074,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1911}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7779","reported_seq":8008,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.837638+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:44.837638+0000","last_peered":"2024-04-26T02:30:44.837638+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:44.837638+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7779,"log_dups_size":0,"ondisk_log_size":7779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31275270,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1920}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'7994","reported_seq":8211,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.777015+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:44.777015+0000","last_peered":"2024-04-26T02:30:44.777015+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:44.777015+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31004934,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8125","reported_seq":8365,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:44.637346+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:44.637346+0000","last_peered":"2024-04-26T02:30:44.637346+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:44.637346+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32739852,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8123,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2015}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'7959","reported_seq":12168,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:43.378881+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:43.378881+0000","last_peered":"2024-04-26T02:30:43.378881+0000","last_clean":"2024-04-26T02:30:43.378881+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:43.378881+0000","last_undegraded":"2024-04-26T02:30:43.378881+0000","last_fullsized":"2024-04-26T02:30:43.378881+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31912460,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'7933","reported_seq":8159,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:42.415343+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:42.415343+0000","last_peered":"2024-04-26T02:30:42.415343+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:42.415343+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31650316,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":502234304,"num_objects":61315,"num_object_clones":0,"num_object_copies":245260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126242,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7220,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1003839488,"data_stored":1003839488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126269,"ondisk_log_size":126269,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700180,"kb_used_data":245056,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93671660,"statfs":{"total":96636764160,"available":95919779840,"internally_reserved":0,"allocated":250937344,"data_stored":250733682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":466022096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,4,6,16,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691700,"kb_used_data":245420,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93680140,"statfs":{"total":96636764160,"available":95928463360,"internally_reserved":0,"allocated":251310080,"data_stored":251106418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573864,"kb_used_data":245152,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93797976,"statfs":{"total":96636764160,"available":96049127424,"internally_reserved":0,"allocated":251035648,"data_stored":250831986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694132,"kb_used_data":247400,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93677708,"statfs":{"total":96636764160,"available":95925972992,"internally_reserved":0,"allocated":253337600,"data_stored":253138602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252452864,"data_stored":252452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250499072,"data_stored":250499072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250458112,"data_stored":250458112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250429440,"data_stored":250429440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:51.008 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:52.769 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:52.770 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:52.870 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-04-26T02:30:52.612223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493296832,"num_objects":60224,"num_object_clones":0,"num_object_copies":240896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127333,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7286,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127360,"ondisk_log_size":127360,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686176,"kb_used_data":967672,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374801184,"statfs":{"total":386547056640,"available":383796412416,"internally_reserved":0,"allocated":990896128,"data_stored":990081480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1759692339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,10,12,9,11,8,5,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17793024,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.716957"},"pg_stats":[{"pgid":"1.f","version":"38'7841","reported_seq":8072,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.789500+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:49.789500+0000","last_peered":"2024-04-26T02:30:49.789500+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:49.789500+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30171136,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8017","reported_seq":8242,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.475337+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:47.475337+0000","last_peered":"2024-04-26T02:30:47.475337+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:47.475337+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31248908,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'7912","reported_seq":8080,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.791885+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:49.791885+0000","last_peered":"2024-04-26T02:30:49.791885+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:49.791885+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30456094,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7911,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7754","reported_seq":11779,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:48.146706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:48.146706+0000","last_peered":"2024-04-26T02:30:48.146706+0000","last_clean":"2024-04-26T02:30:48.146706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:48.146706+0000","last_undegraded":"2024-04-26T02:30:48.146706+0000","last_fullsized":"2024-04-26T02:30:48.146706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:52.593985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30519820,"num_objects":3726,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7752,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7876","reported_seq":8106,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.787157+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:49.787157+0000","last_peered":"2024-04-26T02:30:49.787157+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:49.787157+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7876,"log_dups_size":0,"ondisk_log_size":7876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30478860,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7907","reported_seq":8162,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.789556+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:49.789556+0000","last_peered":"2024-04-26T02:30:49.789556+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:49.789556+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30253056,"num_objects":3693,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7989","reported_seq":12241,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:47.595412+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:47.595412+0000","last_peered":"2024-04-26T02:30:47.595412+0000","last_clean":"2024-04-26T02:30:47.595412+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:47.595412+0000","last_undegraded":"2024-04-26T02:30:47.595412+0000","last_fullsized":"2024-04-26T02:30:47.595412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:50.783579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872076,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8171","reported_seq":8423,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.640161+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:47.640161+0000","last_peered":"2024-04-26T02:30:47.640161+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:47.640161+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31273484,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7947","reported_seq":8464,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.806810+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:30:49.806810+0000","last_peered":"2024-04-26T02:30:49.806810+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:49.806810+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7947,"log_dups_size":0,"ondisk_log_size":7947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31065112,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7872","reported_seq":8045,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.634661+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:49.634661+0000","last_peered":"2024-04-26T02:30:49.634661+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:49.634661+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30241292,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1697}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7935","reported_seq":8169,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.538994+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:49.538994+0000","last_peered":"2024-04-26T02:30:49.538994+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:49.538994+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30993170,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7851","reported_seq":8078,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.902110+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:49.902110+0000","last_peered":"2024-04-26T02:30:49.902110+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:49.902110+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30685446,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8067","reported_seq":8284,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.697629+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:49.697629+0000","last_peered":"2024-04-26T02:30:49.697629+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:49.697629+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30406918,"num_objects":3712,"num_object_clones":0,"num_object_copies":14848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8207","reported_seq":8453,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.861624+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:49.861624+0000","last_peered":"2024-04-26T02:30:49.861624+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:49.861624+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32068108,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8019","reported_seq":12227,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:48.365085+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:48.365085+0000","last_peered":"2024-04-26T02:30:48.365085+0000","last_clean":"2024-04-26T02:30:48.365085+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:48.365085+0000","last_undegraded":"2024-04-26T02:30:48.365085+0000","last_fullsized":"2024-04-26T02:30:48.365085+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8019,"log_dups_size":0,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:48.533922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31420940,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'7995","reported_seq":8221,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.512835+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:47.512835+0000","last_peered":"2024-04-26T02:30:47.512835+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:47.512835+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31142412,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493296832,"num_objects":60224,"num_object_clones":0,"num_object_copies":240896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127333,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7286,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987168768,"data_stored":987168768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127360,"ondisk_log_size":127360,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701292,"kb_used_data":240612,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93670548,"statfs":{"total":96636764160,"available":95918641152,"internally_reserved":0,"allocated":246386688,"data_stored":246183026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,6,2,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709560,"kb_used_data":241072,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93662280,"statfs":{"total":96636764160,"available":95910174720,"internally_reserved":0,"allocated":246857728,"data_stored":246654066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563112,"kb_used_data":242712,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93808728,"statfs":{"total":96636764160,"available":96060137472,"internally_reserved":0,"allocated":248537088,"data_stored":248333426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712212,"kb_used_data":243276,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93659628,"statfs":{"total":96636764160,"available":95907459072,"internally_reserved":0,"allocated":249114624,"data_stored":248910962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,3,2,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247939072,"data_stored":247939072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247435264,"data_stored":247435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245960704,"data_stored":245960704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245833728,"data_stored":245833728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:52.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:53.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:53.949 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:53.965 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-04-26T02:30:52.612223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493296832,"num_objects":60224,"num_object_clones":0,"num_object_copies":240896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127333,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7286,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127360,"ondisk_log_size":127360,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2686176,"kb_used_data":967672,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374801184,"statfs":{"total":386547056640,"available":383796412416,"internally_reserved":0,"allocated":990896128,"data_stored":990081480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1759692339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,10,12,9,11,8,5,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17793024,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.716957"},"pg_stats":[{"pgid":"1.f","version":"38'7841","reported_seq":8072,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.789500+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:49.789500+0000","last_peered":"2024-04-26T02:30:49.789500+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:49.789500+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30171136,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8017","reported_seq":8242,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.475337+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:47.475337+0000","last_peered":"2024-04-26T02:30:47.475337+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:47.475337+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31248908,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'7912","reported_seq":8080,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.791885+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:49.791885+0000","last_peered":"2024-04-26T02:30:49.791885+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:49.791885+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30456094,"num_objects":3719,"num_object_clones":0,"num_object_copies":14876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7911,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1735}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7754","reported_seq":11779,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:48.146706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:48.146706+0000","last_peered":"2024-04-26T02:30:48.146706+0000","last_clean":"2024-04-26T02:30:48.146706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:48.146706+0000","last_undegraded":"2024-04-26T02:30:48.146706+0000","last_fullsized":"2024-04-26T02:30:48.146706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:52.593985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30519820,"num_objects":3726,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7752,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7876","reported_seq":8106,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.787157+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:49.787157+0000","last_peered":"2024-04-26T02:30:49.787157+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:49.787157+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7876,"log_dups_size":0,"ondisk_log_size":7876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30478860,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7907","reported_seq":8162,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.789556+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:49.789556+0000","last_peered":"2024-04-26T02:30:49.789556+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:49.789556+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30253056,"num_objects":3693,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'7989","reported_seq":12241,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:47.595412+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:47.595412+0000","last_peered":"2024-04-26T02:30:47.595412+0000","last_clean":"2024-04-26T02:30:47.595412+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:47.595412+0000","last_undegraded":"2024-04-26T02:30:47.595412+0000","last_fullsized":"2024-04-26T02:30:47.595412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:50.783579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872076,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8171","reported_seq":8423,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.640161+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:47.640161+0000","last_peered":"2024-04-26T02:30:47.640161+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:47.640161+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31273484,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'7947","reported_seq":8464,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.806810+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:30:49.806810+0000","last_peered":"2024-04-26T02:30:49.806810+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:49.806810+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7947,"log_dups_size":0,"ondisk_log_size":7947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31065112,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7872","reported_seq":8045,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.634661+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:49.634661+0000","last_peered":"2024-04-26T02:30:49.634661+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:49.634661+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30241292,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1697}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7935","reported_seq":8169,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.538994+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:49.538994+0000","last_peered":"2024-04-26T02:30:49.538994+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:49.538994+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30993170,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7851","reported_seq":8078,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.902110+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:49.902110+0000","last_peered":"2024-04-26T02:30:49.902110+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:49.902110+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30685446,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8067","reported_seq":8284,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.697629+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:49.697629+0000","last_peered":"2024-04-26T02:30:49.697629+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:49.697629+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30406918,"num_objects":3712,"num_object_clones":0,"num_object_copies":14848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8207","reported_seq":8453,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:49.861624+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:49.861624+0000","last_peered":"2024-04-26T02:30:49.861624+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:49.861624+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32068108,"num_objects":3915,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8019","reported_seq":12227,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:48.365085+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:48.365085+0000","last_peered":"2024-04-26T02:30:48.365085+0000","last_clean":"2024-04-26T02:30:48.365085+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:48.365085+0000","last_undegraded":"2024-04-26T02:30:48.365085+0000","last_fullsized":"2024-04-26T02:30:48.365085+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8019,"log_dups_size":0,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:48.533922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31420940,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'7995","reported_seq":8221,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:47.512835+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:47.512835+0000","last_peered":"2024-04-26T02:30:47.512835+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:47.512835+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31142412,"num_objects":3802,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":493296832,"num_objects":60224,"num_object_clones":0,"num_object_copies":240896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127333,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7286,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987168768,"data_stored":987168768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127360,"ondisk_log_size":127360,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701292,"kb_used_data":240612,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93670548,"statfs":{"total":96636764160,"available":95918641152,"internally_reserved":0,"allocated":246386688,"data_stored":246183026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,6,2,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709560,"kb_used_data":241072,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93662280,"statfs":{"total":96636764160,"available":95910174720,"internally_reserved":0,"allocated":246857728,"data_stored":246654066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563112,"kb_used_data":242712,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93808728,"statfs":{"total":96636764160,"available":96060137472,"internally_reserved":0,"allocated":248537088,"data_stored":248333426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712212,"kb_used_data":243276,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93659628,"statfs":{"total":96636764160,"available":95907459072,"internally_reserved":0,"allocated":249114624,"data_stored":248910962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,3,2,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247939072,"data_stored":247939072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247435264,"data_stored":247435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245960704,"data_stored":245960704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245833728,"data_stored":245833728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:30:53.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:30:53.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7841", 'reported_seq': 8072, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.789500+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:49.789500+0000', 'last_peered': '2024-04-26T02:30:49.789500+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:49.789500+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7841, 'log_dups_size': 0, 'ondisk_log_size': 7841, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30171136, 'num_objects': 3683, 'num_object_clones': 0, 'num_object_copies': 14732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7841, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1756}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8017", 'reported_seq': 8242, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:47.475337+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:47.475337+0000', 'last_peered': '2024-04-26T02:30:47.475337+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:47.475337+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8017, 'log_dups_size': 0, 'ondisk_log_size': 8017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31248908, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2003, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8015, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'7912", 'reported_seq': 8080, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.791885+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:49.791885+0000', 'last_peered': '2024-04-26T02:30:49.791885+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:49.791885+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7912, 'log_dups_size': 0, 'ondisk_log_size': 7912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30456094, 'num_objects': 3719, 'num_object_clones': 0, 'num_object_copies': 14876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3719, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7911, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1735}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'7876", 'reported_seq': 8106, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.787157+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:30:49.787157+0000', 'last_peered': '2024-04-26T02:30:49.787157+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:30:49.787157+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7876, 'log_dups_size': 0, 'ondisk_log_size': 7876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30478860, 'num_objects': 3721, 'num_object_clones': 0, 'num_object_copies': 14884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7874, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1792}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'7907", 'reported_seq': 8162, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.789556+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:49.789556+0000', 'last_peered': '2024-04-26T02:30:49.789556+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:49.789556+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7907, 'log_dups_size': 0, 'ondisk_log_size': 7907, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30253056, 'num_objects': 3693, 'num_object_clones': 0, 'num_object_copies': 14772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7907, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:30:53.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8171", 'reported_seq': 8423, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:47.640161+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:47.640161+0000', 'last_peered': '2024-04-26T02:30:47.640161+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:47.640161+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8171, 'log_dups_size': 0, 'ondisk_log_size': 8171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31273484, 'num_objects': 3818, 'num_object_clones': 0, 'num_object_copies': 15272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8169, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'7947", 'reported_seq': 8464, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.806810+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:30:49.806810+0000', 'last_peered': '2024-04-26T02:30:49.806810+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:30:49.806810+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7947, 'log_dups_size': 0, 'ondisk_log_size': 7947, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31065112, 'num_objects': 3793, 'num_object_clones': 0, 'num_object_copies': 15172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1782, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7944, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 228, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2011}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'7872", 'reported_seq': 8045, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.634661+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:49.634661+0000', 'last_peered': '2024-04-26T02:30:49.634661+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:49.634661+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7872, 'log_dups_size': 0, 'ondisk_log_size': 7872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30241292, 'num_objects': 3692, 'num_object_clones': 0, 'num_object_copies': 14768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7870, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1697}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'7935", 'reported_seq': 8169, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.538994+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:49.538994+0000', 'last_peered': '2024-04-26T02:30:49.538994+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:49.538994+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7935, 'log_dups_size': 0, 'ondisk_log_size': 7935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30993170, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7932, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7851", 'reported_seq': 8078, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.902110+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:49.902110+0000', 'last_peered': '2024-04-26T02:30:49.902110+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:49.902110+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7851, 'log_dups_size': 0, 'ondisk_log_size': 7851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30685446, 'num_objects': 3746, 'num_object_clones': 0, 'num_object_copies': 14984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7850, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8067", 'reported_seq': 8284, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.697629+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:49.697629+0000', 'last_peered': '2024-04-26T02:30:49.697629+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:49.697629+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8067, 'log_dups_size': 0, 'ondisk_log_size': 8067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30406918, 'num_objects': 3712, 'num_object_clones': 0, 'num_object_copies': 14848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8066, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:30:53.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:30:53.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'8207", 'reported_seq': 8453, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:49.861624+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:30:49.861624+0000', 'last_peered': '2024-04-26T02:30:49.861624+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:30:49.861624+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8207, 'log_dups_size': 0, 'ondisk_log_size': 8207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32068108, 'num_objects': 3915, 'num_object_clones': 0, 'num_object_copies': 15660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8205, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:30:53.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:30:53.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'7995", 'reported_seq': 8221, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:47.512835+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:47.512835+0000', 'last_peered': '2024-04-26T02:30:47.512835+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:47.512835+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7995, 'log_dups_size': 0, 'ondisk_log_size': 7995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31142412, 'num_objects': 3802, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7993, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:30:53.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:30:55.826 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:55.826 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25480/239160 objects degraded (10.654%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59790,"data_bytes":489741504,"bytes_used":2748362752,"bytes_avail":383798693888,"bytes_total":386547056640,"degraded_objects":25480,"degraded_total":239160,"degraded_ratio":0.10653955510955009,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":35482,"recovering_keys_per_sec":0,"num_objects_recovered":237,"num_bytes_recovered":425984,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:30:56.038 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:30:56.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:30:57.241 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:30:57.241 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:30:57.267 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-04-26T02:30:56.613183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484228288,"num_objects":59117,"num_object_clones":0,"num_object_copies":236468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128440,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7369,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128467,"ondisk_log_size":128467,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2683556,"kb_used_data":948416,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374803804,"statfs":{"total":386547056640,"available":383799095296,"internally_reserved":0,"allocated":971177984,"data_stored":970363336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1776731699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,9,16,13,10,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23412736,"num_objects":-2858,"num_object_clones":0,"num_object_copies":-11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2858,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003169"},"pg_stats":[{"pgid":"1.f","version":"38'7911","reported_seq":8142,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.912669+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:54.912669+0000","last_peered":"2024-04-26T02:30:54.912669+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:54.912669+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8089","reported_seq":8316,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:52.499555+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:52.499555+0000","last_peered":"2024-04-26T02:30:52.499555+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:52.499555+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8089,"log_dups_size":0,"ondisk_log_size":8089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30659084,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8087,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'7971","reported_seq":8140,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.654504+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:54.654504+0000","last_peered":"2024-04-26T02:30:54.654504+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:54.654504+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29972766,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7970,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7824","reported_seq":11849,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:53.532161+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:53.532161+0000","last_peered":"2024-04-26T02:30:53.532161+0000","last_clean":"2024-04-26T02:30:53.532161+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:53.532161+0000","last_undegraded":"2024-04-26T02:30:53.532161+0000","last_fullsized":"2024-04-26T02:30:53.532161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29946380,"num_objects":3656,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'7950","reported_seq":8180,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.876234+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:30:54.876234+0000","last_peered":"2024-04-26T02:30:54.876234+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:30:54.876234+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7950,"log_dups_size":0,"ondisk_log_size":7950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29872652,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7948,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'7970","reported_seq":8224,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.947451+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:54.947451+0000","last_peered":"2024-04-26T02:30:54.947451+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:54.947451+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8047","reported_seq":12299,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:52.563420+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:52.563420+0000","last_peered":"2024-04-26T02:30:52.563420+0000","last_clean":"2024-04-26T02:30:52.563420+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:52.563420+0000","last_undegraded":"2024-04-26T02:30:52.563420+0000","last_fullsized":"2024-04-26T02:30:52.563420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:56.681731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30396940,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8250","reported_seq":8504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:52.600651+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:52.600651+0000","last_peered":"2024-04-26T02:30:52.600651+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:52.600651+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8250,"log_dups_size":0,"ondisk_log_size":8250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30626316,"num_objects":3739,"num_object_clones":0,"num_object_copies":14956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8248,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8018","reported_seq":8695,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:55.007446+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:30:55.007446+0000","last_peered":"2024-04-26T02:30:55.007446+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:30:55.007446+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30483480,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":308,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'7949","reported_seq":8121,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.762370+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:54.762370+0000","last_peered":"2024-04-26T02:30:54.762370+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:54.762370+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7949,"log_dups_size":0,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29610508,"num_objects":3615,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7947,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'7997","reported_seq":8231,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:55.009828+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:55.009828+0000","last_peered":"2024-04-26T02:30:55.009828+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:55.009828+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30485266,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1788}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7912","reported_seq":8141,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.875993+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:54.875993+0000","last_peered":"2024-04-26T02:30:54.875993+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:54.875993+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30185734,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8133","reported_seq":8350,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.788255+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:54.788255+0000","last_peered":"2024-04-26T02:30:54.788255+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:54.788255+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29866246,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8277","reported_seq":8522,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:54.932212+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:30:54.932212+0000","last_peered":"2024-04-26T02:30:54.932212+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:30:54.932212+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8277,"log_dups_size":0,"ondisk_log_size":8277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31494668,"num_objects":3845,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1866}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8105","reported_seq":12314,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:53.435383+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:53.435383+0000","last_peered":"2024-04-26T02:30:53.435383+0000","last_clean":"2024-04-26T02:30:53.435383+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:53.435383+0000","last_undegraded":"2024-04-26T02:30:53.435383+0000","last_fullsized":"2024-04-26T02:30:53.435383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:53.611638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30716428,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8064","reported_seq":8291,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:52.630418+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:52.630418+0000","last_peered":"2024-04-26T02:30:52.630418+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:52.630418+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8064,"log_dups_size":0,"ondisk_log_size":8064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577164,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":484228288,"num_objects":59117,"num_object_clones":0,"num_object_copies":236468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128440,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7369,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":968982528,"data_stored":968982528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128467,"ondisk_log_size":128467,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117076,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701040,"kb_used_data":236200,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93670800,"statfs":{"total":96636764160,"available":95918899200,"internally_reserved":0,"allocated":241868800,"data_stored":241665138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,3,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709420,"kb_used_data":236772,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93662420,"statfs":{"total":96636764160,"available":95910318080,"internally_reserved":0,"allocated":242454528,"data_stored":242250866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561976,"kb_used_data":237424,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93809864,"statfs":{"total":96636764160,"available":96061300736,"internally_reserved":0,"allocated":243122176,"data_stored":242918514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711120,"kb_used_data":238020,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93660720,"statfs":{"total":96636764160,"available":95908577280,"internally_reserved":0,"allocated":243732480,"data_stored":243528818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,11,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243339264,"data_stored":243339264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242749440,"data_stored":242749440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241557504,"data_stored":241557504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241336320,"data_stored":241336320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:00.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:02.039 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:02.040 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:02.150 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-04-26T02:31:00.618542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475184320,"num_objects":58013,"num_object_clones":0,"num_object_copies":232052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129544,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7432,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129571,"ondisk_log_size":129571,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657636,"kb_used_data":930820,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374829724,"statfs":{"total":386547056640,"available":383825637376,"internally_reserved":0,"allocated":953159680,"data_stored":952345032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1768212019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,6,5,18,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23912448,"num_objects":-2919,"num_object_clones":0,"num_object_copies":-11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007375"},"pg_stats":[{"pgid":"1.f","version":"38'7970","reported_seq":8201,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.855614+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:59.855614+0000","last_peered":"2024-04-26T02:30:59.855614+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:59.855614+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29114368,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8164","reported_seq":8391,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.636710+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:57.636710+0000","last_peered":"2024-04-26T02:30:57.636710+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:57.636710+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30044684,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8037","reported_seq":8206,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.606526+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:59.606526+0000","last_peered":"2024-04-26T02:30:59.606526+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:59.606526+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8037,"log_dups_size":0,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29432094,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8036,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7888","reported_seq":11913,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:58.578640+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:58.578640+0000","last_peered":"2024-04-26T02:30:58.578640+0000","last_clean":"2024-04-26T02:30:58.578640+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:58.578640+0000","last_undegraded":"2024-04-26T02:30:58.578640+0000","last_fullsized":"2024-04-26T02:30:58.578640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:03.574957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422092,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8019","reported_seq":8249,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.037466+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:00.037466+0000","last_peered":"2024-04-26T02:31:00.037466+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:00.037466+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8019,"log_dups_size":0,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29307404,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8047","reported_seq":8301,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.955522+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:59.955522+0000","last_peered":"2024-04-26T02:30:59.955522+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:59.955522+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29106176,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8116","reported_seq":12368,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:57.668552+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:57.668552+0000","last_peered":"2024-04-26T02:30:57.668552+0000","last_clean":"2024-04-26T02:30:57.668552+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:57.668552+0000","last_undegraded":"2024-04-26T02:30:57.668552+0000","last_fullsized":"2024-04-26T02:30:57.668552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8116,"log_dups_size":0,"ondisk_log_size":8116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:02.665287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29831692,"num_objects":3642,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8329","reported_seq":8583,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.685927+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:57.685927+0000","last_peered":"2024-04-26T02:30:57.685927+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:57.685927+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979148,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8089","reported_seq":8891,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.036648+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:00.036648+0000","last_peered":"2024-04-26T02:31:00.036648+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:00.036648+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8089,"log_dups_size":0,"ondisk_log_size":8089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29901848,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8021","reported_seq":8194,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.737959+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:59.737959+0000","last_peered":"2024-04-26T02:30:59.737959+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:59.737959+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8021,"log_dups_size":0,"ondisk_log_size":8021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29020684,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8019,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8047","reported_seq":8281,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.660928+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:59.660928+0000","last_peered":"2024-04-26T02:30:59.660928+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:59.660928+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075666,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7992","reported_seq":8221,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.827153+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:59.827153+0000","last_peered":"2024-04-26T02:30:59.827153+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:59.827153+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29530374,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8193","reported_seq":8410,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.580332+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:59.580332+0000","last_peered":"2024-04-26T02:30:59.580332+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:59.580332+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8193,"log_dups_size":0,"ondisk_log_size":8193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29374726,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8192,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1541}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8351","reported_seq":8598,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.059170+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:00.059170+0000","last_peered":"2024-04-26T02:31:00.059170+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:00.059170+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30888460,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1792}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8169","reported_seq":12378,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:58.494602+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:58.494602+0000","last_peered":"2024-04-26T02:30:58.494602+0000","last_clean":"2024-04-26T02:30:58.494602+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:58.494602+0000","last_undegraded":"2024-04-26T02:30:58.494602+0000","last_fullsized":"2024-04-26T02:30:58.494602+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8169,"log_dups_size":0,"ondisk_log_size":8169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:59.634682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192140,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8139","reported_seq":8368,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.670399+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:57.670399+0000","last_peered":"2024-04-26T02:30:57.670399+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:57.670399+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29962764,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475184320,"num_objects":58013,"num_object_clones":0,"num_object_copies":232052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129544,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7432,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":950808576,"data_stored":950808576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129571,"ondisk_log_size":129571,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704852,"kb_used_data":231692,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93666988,"statfs":{"total":96636764160,"available":95914995712,"internally_reserved":0,"allocated":237252608,"data_stored":237048946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,11,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696636,"kb_used_data":232308,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93675204,"statfs":{"total":96636764160,"available":95923408896,"internally_reserved":0,"allocated":237883392,"data_stored":237679730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561784,"kb_used_data":233072,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93810056,"statfs":{"total":96636764160,"available":96061497344,"internally_reserved":0,"allocated":238665728,"data_stored":238462066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215194,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694364,"kb_used_data":233748,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93677476,"statfs":{"total":96636764160,"available":95925735424,"internally_reserved":0,"allocated":239357952,"data_stored":239154290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238895104,"data_stored":238895104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238231552,"data_stored":238231552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236969984,"data_stored":236969984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236711936,"data_stored":236711936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:02.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:03.930 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:03.932 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:04.055 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-26T02:31:02.619063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475184320,"num_objects":58013,"num_object_clones":0,"num_object_copies":232052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129544,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7432,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129571,"ondisk_log_size":129571,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657636,"kb_used_data":930820,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374829724,"statfs":{"total":386547056640,"available":383825637376,"internally_reserved":0,"allocated":953159680,"data_stored":952345032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1768212019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,6,5,18,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18112512,"num_objects":-2211,"num_object_clones":0,"num_object_copies":-8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007367"},"pg_stats":[{"pgid":"1.f","version":"38'7970","reported_seq":8201,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.855614+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:30:59.855614+0000","last_peered":"2024-04-26T02:30:59.855614+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:30:59.855614+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29114368,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8164","reported_seq":8391,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.636710+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:30:57.636710+0000","last_peered":"2024-04-26T02:30:57.636710+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:30:57.636710+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30044684,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8037","reported_seq":8206,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.606526+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:30:59.606526+0000","last_peered":"2024-04-26T02:30:59.606526+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:30:59.606526+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8037,"log_dups_size":0,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29432094,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8036,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7888","reported_seq":11913,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:58.578640+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:30:58.578640+0000","last_peered":"2024-04-26T02:30:58.578640+0000","last_clean":"2024-04-26T02:30:58.578640+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:30:58.578640+0000","last_undegraded":"2024-04-26T02:30:58.578640+0000","last_fullsized":"2024-04-26T02:30:58.578640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:03.574957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422092,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8019","reported_seq":8249,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.037466+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:00.037466+0000","last_peered":"2024-04-26T02:31:00.037466+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:00.037466+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8019,"log_dups_size":0,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29307404,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8047","reported_seq":8301,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.955522+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:30:59.955522+0000","last_peered":"2024-04-26T02:30:59.955522+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:30:59.955522+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29106176,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8116","reported_seq":12368,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:57.668552+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:30:57.668552+0000","last_peered":"2024-04-26T02:30:57.668552+0000","last_clean":"2024-04-26T02:30:57.668552+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:30:57.668552+0000","last_undegraded":"2024-04-26T02:30:57.668552+0000","last_fullsized":"2024-04-26T02:30:57.668552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8116,"log_dups_size":0,"ondisk_log_size":8116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:02.665287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29831692,"num_objects":3642,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8329","reported_seq":8583,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.685927+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:30:57.685927+0000","last_peered":"2024-04-26T02:30:57.685927+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:30:57.685927+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29979148,"num_objects":3660,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8089","reported_seq":8891,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.036648+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:00.036648+0000","last_peered":"2024-04-26T02:31:00.036648+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:00.036648+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8089,"log_dups_size":0,"ondisk_log_size":8089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29901848,"num_objects":3651,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8021","reported_seq":8194,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.737959+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:30:59.737959+0000","last_peered":"2024-04-26T02:30:59.737959+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:30:59.737959+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8021,"log_dups_size":0,"ondisk_log_size":8021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29020684,"num_objects":3543,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8019,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8047","reported_seq":8281,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.660928+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:30:59.660928+0000","last_peered":"2024-04-26T02:30:59.660928+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:30:59.660928+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075666,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'7992","reported_seq":8221,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.827153+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:30:59.827153+0000","last_peered":"2024-04-26T02:30:59.827153+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:30:59.827153+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29530374,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8193","reported_seq":8410,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:59.580332+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:30:59.580332+0000","last_peered":"2024-04-26T02:30:59.580332+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:30:59.580332+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8193,"log_dups_size":0,"ondisk_log_size":8193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29374726,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8192,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1541}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8351","reported_seq":8598,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:00.059170+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:00.059170+0000","last_peered":"2024-04-26T02:31:00.059170+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:00.059170+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30888460,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1792}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8169","reported_seq":12378,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:30:58.494602+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:30:58.494602+0000","last_peered":"2024-04-26T02:30:58.494602+0000","last_clean":"2024-04-26T02:30:58.494602+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:30:58.494602+0000","last_undegraded":"2024-04-26T02:30:58.494602+0000","last_fullsized":"2024-04-26T02:30:58.494602+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8169,"log_dups_size":0,"ondisk_log_size":8169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:59.634682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192140,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8139","reported_seq":8368,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:30:57.670399+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:30:57.670399+0000","last_peered":"2024-04-26T02:30:57.670399+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:30:57.670399+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29962764,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":475184320,"num_objects":58013,"num_object_clones":0,"num_object_copies":232052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129544,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7432,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":950808576,"data_stored":950808576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129571,"ondisk_log_size":129571,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704852,"kb_used_data":231692,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93666988,"statfs":{"total":96636764160,"available":95914995712,"internally_reserved":0,"allocated":237252608,"data_stored":237048946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,11,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696636,"kb_used_data":232308,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93675204,"statfs":{"total":96636764160,"available":95923408896,"internally_reserved":0,"allocated":237883392,"data_stored":237679730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561784,"kb_used_data":233072,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93810056,"statfs":{"total":96636764160,"available":96061497344,"internally_reserved":0,"allocated":238665728,"data_stored":238462066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215194,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694364,"kb_used_data":233748,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93677476,"statfs":{"total":96636764160,"available":95925735424,"internally_reserved":0,"allocated":239357952,"data_stored":239154290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238895104,"data_stored":238895104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238231552,"data_stored":238231552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236969984,"data_stored":236969984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236711936,"data_stored":236711936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'7970", 'reported_seq': 8201, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.855614+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:30:59.855614+0000', 'last_peered': '2024-04-26T02:30:59.855614+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:30:59.855614+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7970, 'log_dups_size': 0, 'ondisk_log_size': 7970, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29114368, 'num_objects': 3554, 'num_object_clones': 0, 'num_object_copies': 14216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7970, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8164", 'reported_seq': 8391, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:57.636710+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:30:57.636710+0000', 'last_peered': '2024-04-26T02:30:57.636710+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:30:57.636710+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8164, 'log_dups_size': 0, 'ondisk_log_size': 8164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30044684, 'num_objects': 3668, 'num_object_clones': 0, 'num_object_copies': 14672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8162, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8037", 'reported_seq': 8206, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.606526+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:30:59.606526+0000', 'last_peered': '2024-04-26T02:30:59.606526+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:30:59.606526+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8037, 'log_dups_size': 0, 'ondisk_log_size': 8037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29432094, 'num_objects': 3594, 'num_object_clones': 0, 'num_object_copies': 14376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3594, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8036, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1610}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8019", 'reported_seq': 8249, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:00.037466+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:00.037466+0000', 'last_peered': '2024-04-26T02:31:00.037466+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:00.037466+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8019, 'log_dups_size': 0, 'ondisk_log_size': 8019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29307404, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8017, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:04.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8047", 'reported_seq': 8301, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.955522+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:30:59.955522+0000', 'last_peered': '2024-04-26T02:30:59.955522+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:30:59.955522+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8047, 'log_dups_size': 0, 'ondisk_log_size': 8047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29106176, 'num_objects': 3553, 'num_object_clones': 0, 'num_object_copies': 14212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8047, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8329", 'reported_seq': 8583, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:57.685927+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:30:57.685927+0000', 'last_peered': '2024-04-26T02:30:57.685927+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:30:57.685927+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8329, 'log_dups_size': 0, 'ondisk_log_size': 8329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29979148, 'num_objects': 3660, 'num_object_clones': 0, 'num_object_copies': 14640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8327, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8089", 'reported_seq': 8891, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:00.036648+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:00.036648+0000', 'last_peered': '2024-04-26T02:31:00.036648+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:00.036648+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8089, 'log_dups_size': 0, 'ondisk_log_size': 8089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29901848, 'num_objects': 3651, 'num_object_clones': 0, 'num_object_copies': 14604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1639, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8086, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 371, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2012}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8021", 'reported_seq': 8194, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.737959+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:30:59.737959+0000', 'last_peered': '2024-04-26T02:30:59.737959+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:30:59.737959+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8021, 'log_dups_size': 0, 'ondisk_log_size': 8021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29020684, 'num_objects': 3543, 'num_object_clones': 0, 'num_object_copies': 14172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8019, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1548}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8047", 'reported_seq': 8281, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.660928+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:30:59.660928+0000', 'last_peered': '2024-04-26T02:30:59.660928+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:30:59.660928+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8047, 'log_dups_size': 0, 'ondisk_log_size': 8047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30075666, 'num_objects': 3672, 'num_object_clones': 0, 'num_object_copies': 14688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8044, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:04.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'7992", 'reported_seq': 8221, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.827153+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:30:59.827153+0000', 'last_peered': '2024-04-26T02:30:59.827153+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:30:59.827153+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 7992, 'log_dups_size': 0, 'ondisk_log_size': 7992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29530374, 'num_objects': 3605, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7991, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8193", 'reported_seq': 8410, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:59.580332+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:30:59.580332+0000', 'last_peered': '2024-04-26T02:30:59.580332+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:30:59.580332+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8193, 'log_dups_size': 0, 'ondisk_log_size': 8193, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29374726, 'num_objects': 3586, 'num_object_clones': 0, 'num_object_copies': 14344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8192, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'8351", 'reported_seq': 8598, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:00.059170+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:00.059170+0000', 'last_peered': '2024-04-26T02:31:00.059170+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:00.059170+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8351, 'log_dups_size': 0, 'ondisk_log_size': 8351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30888460, 'num_objects': 3771, 'num_object_clones': 0, 'num_object_copies': 15084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8349, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:04.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8139", 'reported_seq': 8368, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:30:57.670399+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:30:57.670399+0000', 'last_peered': '2024-04-26T02:30:57.670399+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:30:57.670399+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8139, 'log_dups_size': 0, 'ondisk_log_size': 8139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29962764, 'num_objects': 3658, 'num_object_clones': 0, 'num_object_copies': 14632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8137, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:04.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:06.104 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:06.104 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25337/230168 objects degraded (11.008%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":462,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57542,"data_bytes":471325888,"bytes_used":2712276992,"bytes_avail":383834779648,"bytes_total":386547056640,"degraded_objects":25337,"degraded_total":230168,"degraded_ratio":0.11008046296618122,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:06.190 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:06.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:07.364 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:07.365 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:07.580 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-04-26T02:31:06.621826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465697984,"num_objects":56855,"num_object_clones":0,"num_object_copies":227420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130702,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130729,"ondisk_log_size":130729,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2631376,"kb_used_data":912864,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374855984,"statfs":{"total":386547056640,"available":383852527616,"internally_reserved":0,"allocated":934772736,"data_stored":933958088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1759692339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,5,10,6,5,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24043520,"num_objects":-2935,"num_object_clones":0,"num_object_copies":-11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009139"},"pg_stats":[{"pgid":"1.f","version":"38'8035","reported_seq":8267,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.117706+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:05.117706+0000","last_peered":"2024-04-26T02:31:05.117706+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:05.117706+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8035,"log_dups_size":0,"ondisk_log_size":8035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28581888,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8248","reported_seq":8474,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:02.901548+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:02.901548+0000","last_peered":"2024-04-26T02:31:02.901548+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:02.901548+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29356556,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8103","reported_seq":8271,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:04.734246+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:04.734246+0000","last_peered":"2024-04-26T02:31:04.734246+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:04.734246+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28891422,"num_objects":3528,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8102,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'7964","reported_seq":11989,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:03.526794+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:03.526794+0000","last_peered":"2024-04-26T02:31:03.526794+0000","last_clean":"2024-04-26T02:31:03.526794+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:03.526794+0000","last_undegraded":"2024-04-26T02:31:03.526794+0000","last_fullsized":"2024-04-26T02:31:03.526794+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28799500,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7962,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8086","reported_seq":8316,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:04.923370+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:04.923370+0000","last_peered":"2024-04-26T02:31:04.923370+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:04.923370+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28758540,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8124","reported_seq":8379,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.128358+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:05.128358+0000","last_peered":"2024-04-26T02:31:05.128358+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:05.128358+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28475392,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8195","reported_seq":12447,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:02.826162+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:02.826162+0000","last_peered":"2024-04-26T02:31:02.826162+0000","last_clean":"2024-04-26T02:31:02.826162+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:02.826162+0000","last_undegraded":"2024-04-26T02:31:02.826162+0000","last_fullsized":"2024-04-26T02:31:02.826162+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29184524,"num_objects":3563,"num_object_clones":0,"num_object_copies":14252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8415","reported_seq":8668,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:02.842068+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:02.842068+0000","last_peered":"2024-04-26T02:31:02.842068+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:02.842068+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29274636,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8162","reported_seq":9058,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.070459+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:05.070459+0000","last_peered":"2024-04-26T02:31:05.070459+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:05.070459+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8162,"log_dups_size":0,"ondisk_log_size":8162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29303832,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8075","reported_seq":8248,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:04.748083+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:04.748083+0000","last_peered":"2024-04-26T02:31:04.748083+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:04.748083+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28578316,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1494}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8118","reported_seq":8352,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.119326+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:05.119326+0000","last_peered":"2024-04-26T02:31:05.119326+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:05.119326+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8118,"log_dups_size":0,"ondisk_log_size":8118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29494034,"num_objects":3601,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8115,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8065","reported_seq":8294,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.151018+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:05.151018+0000","last_peered":"2024-04-26T02:31:05.151018+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:05.151018+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8065,"log_dups_size":0,"ondisk_log_size":8065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28932358,"num_objects":3532,"num_object_clones":0,"num_object_copies":14128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1634}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8257","reported_seq":8474,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:04.705473+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:04.705473+0000","last_peered":"2024-04-26T02:31:04.705473+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:04.705473+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28850438,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8428","reported_seq":8675,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:05.004125+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:05.004125+0000","last_peered":"2024-04-26T02:31:05.004125+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:05.004125+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30257676,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8426,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1715}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8232","reported_seq":12442,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:03.757689+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:03.757689+0000","last_peered":"2024-04-26T02:31:03.757689+0000","last_clean":"2024-04-26T02:31:03.757689+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:03.757689+0000","last_undegraded":"2024-04-26T02:31:03.757689+0000","last_fullsized":"2024-04-26T02:31:03.757689+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:05.494102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29676044,"num_objects":3623,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8222","reported_seq":8451,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:02.719316+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:02.719316+0000","last_peered":"2024-04-26T02:31:02.719316+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:02.719316+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29282828,"num_objects":3575,"num_object_clones":0,"num_object_copies":14300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465697984,"num_objects":56855,"num_object_clones":0,"num_object_copies":227420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130702,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":932089856,"data_stored":932089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130729,"ondisk_log_size":130729,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687868,"kb_used_data":227188,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93683972,"statfs":{"total":96636764160,"available":95932387328,"internally_reserved":0,"allocated":232640512,"data_stored":232436850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,7,1,3,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696288,"kb_used_data":227800,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93675552,"statfs":{"total":96636764160,"available":95923765248,"internally_reserved":0,"allocated":233267200,"data_stored":233063538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549104,"kb_used_data":228700,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93822736,"statfs":{"total":96636764160,"available":96074481664,"internally_reserved":0,"allocated":234188800,"data_stored":233985138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698116,"kb_used_data":229176,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93673724,"statfs":{"total":96636764160,"available":95921893376,"internally_reserved":0,"allocated":234676224,"data_stored":234472562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234115072,"data_stored":234115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233553920,"data_stored":233553920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232361984,"data_stored":232361984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232058880,"data_stored":232058880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:10.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:12.600 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:12.600 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:12.667 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-04-26T02:31:10.622845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":456514752,"num_objects":55734,"num_object_clones":0,"num_object_copies":222936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131823,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7522,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131850,"ondisk_log_size":131850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630748,"kb_used_data":895600,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374856612,"statfs":{"total":386547056640,"available":383853170688,"internally_reserved":0,"allocated":917094400,"data_stored":916279752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1776731699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,6,9,10,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25272320,"num_objects":-3085,"num_object_clones":0,"num_object_copies":-12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009140"},"pg_stats":[{"pgid":"1.f","version":"38'8111","reported_seq":8343,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.237163+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:10.237163+0000","last_peered":"2024-04-26T02:31:10.237163+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:10.237163+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27959296,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8321","reported_seq":8549,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.867283+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:07.867283+0000","last_peered":"2024-04-26T02:31:07.867283+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:07.867283+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8321,"log_dups_size":0,"ondisk_log_size":8321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28758540,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8160","reported_seq":8329,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.786658+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:09.786658+0000","last_peered":"2024-04-26T02:31:09.786658+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:09.786658+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28424478,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8159,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8030","reported_seq":12055,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:08.808385+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:08.808385+0000","last_peered":"2024-04-26T02:31:08.808385+0000","last_clean":"2024-04-26T02:31:08.808385+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:08.808385+0000","last_undegraded":"2024-04-26T02:31:08.808385+0000","last_fullsized":"2024-04-26T02:31:08.808385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:09.444098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28258828,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8158","reported_seq":8388,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.235351+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:10.235351+0000","last_peered":"2024-04-26T02:31:10.235351+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:10.235351+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28168716,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8156,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8191","reported_seq":8446,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.240665+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:10.240665+0000","last_peered":"2024-04-26T02:31:10.240665+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:10.240665+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27926528,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8191,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8274","reported_seq":12526,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:07.835076+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:07.835076+0000","last_peered":"2024-04-26T02:31:07.835076+0000","last_clean":"2024-04-26T02:31:07.835076+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:07.835076+0000","last_undegraded":"2024-04-26T02:31:07.835076+0000","last_fullsized":"2024-04-26T02:31:07.835076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:08.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28537356,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8480","reported_seq":8734,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.706229+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:07.706229+0000","last_peered":"2024-04-26T02:31:07.706229+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:07.706229+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742156,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8226","reported_seq":9210,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.138378+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:10.138378+0000","last_peered":"2024-04-26T02:31:10.138378+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:10.138378+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28779544,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8223,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8144","reported_seq":8317,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.880910+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:09.880910+0000","last_peered":"2024-04-26T02:31:09.880910+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:09.880910+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8144,"log_dups_size":0,"ondisk_log_size":8144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013068,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8186","reported_seq":8420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.078347+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:10.078347+0000","last_peered":"2024-04-26T02:31:10.078347+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:10.078347+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8186,"log_dups_size":0,"ondisk_log_size":8186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28936978,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8142","reported_seq":8371,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.300537+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:10.300537+0000","last_peered":"2024-04-26T02:31:10.300537+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:10.300537+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301574,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1557}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8326","reported_seq":8542,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.523728+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:09.523728+0000","last_peered":"2024-04-26T02:31:09.523728+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:09.523728+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285190,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8513","reported_seq":8760,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.131421+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:10.131421+0000","last_peered":"2024-04-26T02:31:10.131421+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:10.131421+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29561356,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8306","reported_seq":12515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:08.718085+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:08.718085+0000","last_peered":"2024-04-26T02:31:08.718085+0000","last_clean":"2024-04-26T02:31:08.718085+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:08.718085+0000","last_undegraded":"2024-04-26T02:31:08.718085+0000","last_fullsized":"2024-04-26T02:31:08.718085+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:11.383349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29069836,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8282","reported_seq":8511,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.788514+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:07.788514+0000","last_peered":"2024-04-26T02:31:07.788514+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:07.788514+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28791308,"num_objects":3515,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":456514752,"num_objects":55734,"num_object_clones":0,"num_object_copies":222936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131823,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7522,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913772544,"data_stored":913772544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131850,"ondisk_log_size":131850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687656,"kb_used_data":222816,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93684184,"statfs":{"total":96636764160,"available":95932604416,"internally_reserved":0,"allocated":228163584,"data_stored":227959922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696100,"kb_used_data":223452,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93675740,"statfs":{"total":96636764160,"available":95923957760,"internally_reserved":0,"allocated":228814848,"data_stored":228611186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548940,"kb_used_data":224380,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93822900,"statfs":{"total":96636764160,"available":96074649600,"internally_reserved":0,"allocated":229765120,"data_stored":229561458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698052,"kb_used_data":224952,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93673788,"statfs":{"total":96636764160,"available":95921958912,"internally_reserved":0,"allocated":230350848,"data_stored":230147186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229662720,"data_stored":229662720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228962304,"data_stored":228962304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227799040,"data_stored":227799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":227348480,"data_stored":227348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:12.668 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:14.286 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:14.386 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-04-26T02:31:12.623357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":456514752,"num_objects":55734,"num_object_clones":0,"num_object_copies":222936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131823,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7522,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131850,"ondisk_log_size":131850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630748,"kb_used_data":895600,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374856612,"statfs":{"total":386547056640,"available":383853170688,"internally_reserved":0,"allocated":917094400,"data_stored":916279752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1776731699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,6,9,10,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18669568,"num_objects":-2279,"num_object_clones":0,"num_object_copies":-9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004815"},"pg_stats":[{"pgid":"1.f","version":"38'8111","reported_seq":8343,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.237163+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:10.237163+0000","last_peered":"2024-04-26T02:31:10.237163+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:10.237163+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27959296,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8321","reported_seq":8549,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.867283+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:07.867283+0000","last_peered":"2024-04-26T02:31:07.867283+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:07.867283+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8321,"log_dups_size":0,"ondisk_log_size":8321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28758540,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8160","reported_seq":8329,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.786658+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:09.786658+0000","last_peered":"2024-04-26T02:31:09.786658+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:09.786658+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28424478,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8159,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8030","reported_seq":12055,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:08.808385+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:08.808385+0000","last_peered":"2024-04-26T02:31:08.808385+0000","last_clean":"2024-04-26T02:31:08.808385+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:08.808385+0000","last_undegraded":"2024-04-26T02:31:08.808385+0000","last_fullsized":"2024-04-26T02:31:08.808385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:09.444098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28258828,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8158","reported_seq":8388,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.235351+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:10.235351+0000","last_peered":"2024-04-26T02:31:10.235351+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:10.235351+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28168716,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8156,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8191","reported_seq":8446,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.240665+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:10.240665+0000","last_peered":"2024-04-26T02:31:10.240665+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:10.240665+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27926528,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8191,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8274","reported_seq":12526,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:07.835076+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:07.835076+0000","last_peered":"2024-04-26T02:31:07.835076+0000","last_clean":"2024-04-26T02:31:07.835076+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:07.835076+0000","last_undegraded":"2024-04-26T02:31:07.835076+0000","last_fullsized":"2024-04-26T02:31:07.835076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:08.624897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28537356,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8480","reported_seq":8734,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.706229+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:07.706229+0000","last_peered":"2024-04-26T02:31:07.706229+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:07.706229+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28742156,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8226","reported_seq":9210,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.138378+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:10.138378+0000","last_peered":"2024-04-26T02:31:10.138378+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:10.138378+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28779544,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8223,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8144","reported_seq":8317,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.880910+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:09.880910+0000","last_peered":"2024-04-26T02:31:09.880910+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:09.880910+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8144,"log_dups_size":0,"ondisk_log_size":8144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013068,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8186","reported_seq":8420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.078347+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:10.078347+0000","last_peered":"2024-04-26T02:31:10.078347+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:10.078347+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8186,"log_dups_size":0,"ondisk_log_size":8186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28936978,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8142","reported_seq":8371,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.300537+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:10.300537+0000","last_peered":"2024-04-26T02:31:10.300537+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:10.300537+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28301574,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1557}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8326","reported_seq":8542,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:09.523728+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:09.523728+0000","last_peered":"2024-04-26T02:31:09.523728+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:09.523728+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28285190,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8513","reported_seq":8760,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:10.131421+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:10.131421+0000","last_peered":"2024-04-26T02:31:10.131421+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:10.131421+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29561356,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8306","reported_seq":12515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:08.718085+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:08.718085+0000","last_peered":"2024-04-26T02:31:08.718085+0000","last_clean":"2024-04-26T02:31:08.718085+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:08.718085+0000","last_undegraded":"2024-04-26T02:31:08.718085+0000","last_fullsized":"2024-04-26T02:31:08.718085+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:11.383349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29069836,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8282","reported_seq":8511,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:07.788514+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:07.788514+0000","last_peered":"2024-04-26T02:31:07.788514+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:07.788514+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28791308,"num_objects":3515,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":456514752,"num_objects":55734,"num_object_clones":0,"num_object_copies":222936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131823,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7522,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913772544,"data_stored":913772544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131850,"ondisk_log_size":131850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687656,"kb_used_data":222816,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93684184,"statfs":{"total":96636764160,"available":95932604416,"internally_reserved":0,"allocated":228163584,"data_stored":227959922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696100,"kb_used_data":223452,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93675740,"statfs":{"total":96636764160,"available":95923957760,"internally_reserved":0,"allocated":228814848,"data_stored":228611186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548940,"kb_used_data":224380,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93822900,"statfs":{"total":96636764160,"available":96074649600,"internally_reserved":0,"allocated":229765120,"data_stored":229561458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698052,"kb_used_data":224952,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93673788,"statfs":{"total":96636764160,"available":95921958912,"internally_reserved":0,"allocated":230350848,"data_stored":230147186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229662720,"data_stored":229662720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228962304,"data_stored":228962304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227799040,"data_stored":227799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":227348480,"data_stored":227348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8111", 'reported_seq': 8343, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.237163+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:31:10.237163+0000', 'last_peered': '2024-04-26T02:31:10.237163+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:31:10.237163+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8111, 'log_dups_size': 0, 'ondisk_log_size': 8111, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27959296, 'num_objects': 3413, 'num_object_clones': 0, 'num_object_copies': 13652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8111, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8321", 'reported_seq': 8549, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:07.867283+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:31:07.867283+0000', 'last_peered': '2024-04-26T02:31:07.867283+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:31:07.867283+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8321, 'log_dups_size': 0, 'ondisk_log_size': 8321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28758540, 'num_objects': 3511, 'num_object_clones': 0, 'num_object_copies': 14044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8319, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8160", 'reported_seq': 8329, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:09.786658+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:31:09.786658+0000', 'last_peered': '2024-04-26T02:31:09.786658+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:31:09.786658+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28424478, 'num_objects': 3471, 'num_object_clones': 0, 'num_object_copies': 13884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3471, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8159, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1487}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:14.387 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8158", 'reported_seq': 8388, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.235351+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:10.235351+0000', 'last_peered': '2024-04-26T02:31:10.235351+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:10.235351+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8158, 'log_dups_size': 0, 'ondisk_log_size': 8158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28168716, 'num_objects': 3439, 'num_object_clones': 0, 'num_object_copies': 13756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8156, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1510}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8191", 'reported_seq': 8446, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.240665+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:31:10.240665+0000', 'last_peered': '2024-04-26T02:31:10.240665+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:31:10.240665+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8191, 'log_dups_size': 0, 'ondisk_log_size': 8191, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27926528, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8191, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1459}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8480", 'reported_seq': 8734, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:07.706229+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:31:07.706229+0000', 'last_peered': '2024-04-26T02:31:07.706229+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:31:07.706229+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8480, 'log_dups_size': 0, 'ondisk_log_size': 8480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28742156, 'num_objects': 3509, 'num_object_clones': 0, 'num_object_copies': 14036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8478, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1457}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8226", 'reported_seq': 9210, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.138378+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:10.138378+0000', 'last_peered': '2024-04-26T02:31:10.138378+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:10.138378+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8226, 'log_dups_size': 0, 'ondisk_log_size': 8226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28779544, 'num_objects': 3514, 'num_object_clones': 0, 'num_object_copies': 14056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1549, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8223, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1965}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8144", 'reported_seq': 8317, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:09.880910+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:31:09.880910+0000', 'last_peered': '2024-04-26T02:31:09.880910+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:31:09.880910+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8144, 'log_dups_size': 0, 'ondisk_log_size': 8144, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013068, 'num_objects': 3420, 'num_object_clones': 0, 'num_object_copies': 13680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8142, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1425}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:14.388 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8186", 'reported_seq': 8420, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.078347+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:31:10.078347+0000', 'last_peered': '2024-04-26T02:31:10.078347+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:31:10.078347+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8186, 'log_dups_size': 0, 'ondisk_log_size': 8186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28936978, 'num_objects': 3533, 'num_object_clones': 0, 'num_object_copies': 14132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8183, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8142", 'reported_seq': 8371, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.300537+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:31:10.300537+0000', 'last_peered': '2024-04-26T02:31:10.300537+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:31:10.300537+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8142, 'log_dups_size': 0, 'ondisk_log_size': 8142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28301574, 'num_objects': 3455, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8141, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8326", 'reported_seq': 8542, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:09.523728+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:31:09.523728+0000', 'last_peered': '2024-04-26T02:31:09.523728+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:31:09.523728+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8326, 'log_dups_size': 0, 'ondisk_log_size': 8326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28285190, 'num_objects': 3453, 'num_object_clones': 0, 'num_object_copies': 13812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8325, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'8513", 'reported_seq': 8760, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:10.131421+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:10.131421+0000', 'last_peered': '2024-04-26T02:31:10.131421+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:10.131421+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8513, 'log_dups_size': 0, 'ondisk_log_size': 8513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29561356, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8511, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:14.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8282", 'reported_seq': 8511, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:07.788514+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:31:07.788514+0000', 'last_peered': '2024-04-26T02:31:07.788514+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:31:07.788514+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8282, 'log_dups_size': 0, 'ondisk_log_size': 8282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28791308, 'num_objects': 3515, 'num_object_clones': 0, 'num_object_copies': 14060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8280, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:14.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:16.250 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:16.250 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25247/221208 objects degraded (11.413%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":472,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55302,"data_bytes":452975808,"bytes_used":2694602752,"bytes_avail":383852453888,"bytes_total":386547056640,"degraded_objects":25247,"degraded_total":221208,"degraded_ratio":0.11413240027485444,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:16.327 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:16.327 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:18.004 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:18.005 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:18.222 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-26T02:31:16.634545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447413440,"num_objects":54623,"num_object_clones":0,"num_object_copies":218492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132934,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7570,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132961,"ondisk_log_size":132961,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618248,"kb_used_data":877544,"kb_used_omap":14,"kb_used_meta":1740657,"kb_avail":374869112,"statfs":{"total":386547056640,"available":383865970688,"internally_reserved":0,"allocated":898605056,"data_stored":897790408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1782433331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,15,8,7,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23912448,"num_objects":-2919,"num_object_clones":0,"num_object_copies":-11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014820"},"pg_stats":[{"pgid":"1.f","version":"38'8180","reported_seq":8411,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.110721+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:15.110721+0000","last_peered":"2024-04-26T02:31:15.110721+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:15.110721+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":3344,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8391","reported_seq":8619,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:12.886078+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:12.886078+0000","last_peered":"2024-04-26T02:31:12.886078+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:12.886078+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185100,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8220","reported_seq":8389,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:14.675362+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:14.675362+0000","last_peered":"2024-04-26T02:31:14.675362+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:14.675362+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8220,"log_dups_size":0,"ondisk_log_size":8220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27932958,"num_objects":3411,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8219,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8099","reported_seq":12124,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:13.790936+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:13.790936+0000","last_peered":"2024-04-26T02:31:13.790936+0000","last_clean":"2024-04-26T02:31:13.790936+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:13.790936+0000","last_undegraded":"2024-04-26T02:31:13.790936+0000","last_fullsized":"2024-04-26T02:31:13.790936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:15.328199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27693580,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8227","reported_seq":8457,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.230479+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:15.230479+0000","last_peered":"2024-04-26T02:31:15.230479+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:15.230479+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27603468,"num_objects":3370,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8225,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8252","reported_seq":8508,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.156996+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:15.156996+0000","last_peered":"2024-04-26T02:31:15.156996+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:15.156996+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8252,"log_dups_size":0,"ondisk_log_size":8252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27426816,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8252,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8349","reported_seq":12601,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:12.754657+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:12.754657+0000","last_peered":"2024-04-26T02:31:12.754657+0000","last_clean":"2024-04-26T02:31:12.754657+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:12.754657+0000","last_undegraded":"2024-04-26T02:31:12.754657+0000","last_fullsized":"2024-04-26T02:31:12.754657+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:14.508003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27922956,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8548","reported_seq":8799,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:12.811152+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:12.811152+0000","last_peered":"2024-04-26T02:31:12.811152+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:12.811152+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185100,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1389}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8295","reported_seq":9375,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.293910+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:15.293910+0000","last_peered":"2024-04-26T02:31:15.293910+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:15.293910+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28214296,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8292,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8202","reported_seq":8375,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:14.861814+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:14.861814+0000","last_peered":"2024-04-26T02:31:14.861814+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:14.861814+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27537932,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1367}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8253","reported_seq":8487,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.258949+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:15.258949+0000","last_peered":"2024-04-26T02:31:15.258949+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:15.258949+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8253,"log_dups_size":0,"ondisk_log_size":8253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28388114,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8250,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8218","reported_seq":8447,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.342516+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:15.342516+0000","last_peered":"2024-04-26T02:31:15.342516+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:15.342516+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8218,"log_dups_size":0,"ondisk_log_size":8218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27678982,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8405","reported_seq":8622,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:14.911018+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:14.911018+0000","last_peered":"2024-04-26T02:31:14.911018+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:14.911018+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27638022,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8584","reported_seq":8831,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:15.242572+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:15.242572+0000","last_peered":"2024-04-26T02:31:15.242572+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:15.242572+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28979724,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8381","reported_seq":12591,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:13.866742+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:13.866742+0000","last_peered":"2024-04-26T02:31:13.866742+0000","last_clean":"2024-04-26T02:31:13.866742+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:13.866742+0000","last_undegraded":"2024-04-26T02:31:13.866742+0000","last_fullsized":"2024-04-26T02:31:13.866742+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8381,"log_dups_size":0,"ondisk_log_size":8381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:17.385537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28455436,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8357","reported_seq":8586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:12.778996+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:12.778996+0000","last_peered":"2024-04-26T02:31:12.778996+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:12.778996+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28176908,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447413440,"num_objects":54623,"num_object_clones":0,"num_object_copies":218492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132934,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7570,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":895434752,"data_stored":895434752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132961,"ondisk_log_size":132961,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691480,"kb_used_data":218320,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93680360,"statfs":{"total":96636764160,"available":95928688640,"internally_reserved":0,"allocated":223559680,"data_stored":223356018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679076,"kb_used_data":218908,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93692764,"statfs":{"total":96636764160,"available":95941390336,"internally_reserved":0,"allocated":224161792,"data_stored":223958130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,0,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566644,"kb_used_data":219884,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93805196,"statfs":{"total":96636764160,"available":96056520704,"internally_reserved":0,"allocated":225161216,"data_stored":224957554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681048,"kb_used_data":220432,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93690792,"statfs":{"total":96636764160,"available":95939371008,"internally_reserved":0,"allocated":225722368,"data_stored":225518706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225050624,"data_stored":225050624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224276480,"data_stored":224276480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223260672,"data_stored":223260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222846976,"data_stored":222846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:21.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:22.580 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:22.581 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:22.640 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-04-26T02:31:20.635632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":438131904,"num_objects":53490,"num_object_clones":0,"num_object_copies":213960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134067,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134094,"ondisk_log_size":134094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2610496,"kb_used_data":860060,"kb_used_omap":14,"kb_used_meta":1750385,"kb_avail":374876864,"statfs":{"total":386547056640,"available":383873908736,"internally_reserved":0,"allocated":880701440,"data_stored":879886792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1792394803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,7,10,8,9,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25296896,"num_objects":-3088,"num_object_clones":0,"num_object_copies":-12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013282"},"pg_stats":[{"pgid":"1.f","version":"38'8245","reported_seq":8477,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.547123+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:20.547123+0000","last_peered":"2024-04-26T02:31:20.547123+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:20.547123+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26861568,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8471","reported_seq":8699,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:17.988976+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:17.988976+0000","last_peered":"2024-04-26T02:31:17.988976+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:17.988976+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27529740,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8286","reported_seq":8455,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.493353+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:19.493353+0000","last_peered":"2024-04-26T02:31:19.493353+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:19.493353+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8286,"log_dups_size":0,"ondisk_log_size":8286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27392286,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8285,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8165","reported_seq":12190,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:19.074879+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:19.074879+0000","last_peered":"2024-04-26T02:31:19.074879+0000","last_clean":"2024-04-26T02:31:19.074879+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:19.074879+0000","last_undegraded":"2024-04-26T02:31:19.074879+0000","last_fullsized":"2024-04-26T02:31:19.074879+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:20.502916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27152908,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8298","reported_seq":8527,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.467036+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:20.467036+0000","last_peered":"2024-04-26T02:31:20.467036+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:20.467036+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021836,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8325","reported_seq":8581,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.444569+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:20.444569+0000","last_peered":"2024-04-26T02:31:20.444569+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:20.444569+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26828800,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8398","reported_seq":12650,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:18.018626+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:18.018626+0000","last_peered":"2024-04-26T02:31:18.018626+0000","last_clean":"2024-04-26T02:31:18.018626+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:18.018626+0000","last_undegraded":"2024-04-26T02:31:18.018626+0000","last_fullsized":"2024-04-26T02:31:18.018626+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:19.539351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27521548,"num_objects":3360,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8620","reported_seq":8874,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:17.966014+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:17.966014+0000","last_peered":"2024-04-26T02:31:17.966014+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:17.966014+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27595276,"num_objects":3369,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8362","reported_seq":9546,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.492343+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:20.492343+0000","last_peered":"2024-04-26T02:31:20.492343+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:20.492343+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27665432,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8288","reported_seq":8461,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.958351+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:19.958351+0000","last_peered":"2024-04-26T02:31:19.958351+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:19.958351+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8288,"log_dups_size":0,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26833420,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8331","reported_seq":8564,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.504662+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:20.504662+0000","last_peered":"2024-04-26T02:31:20.504662+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:20.504662+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27749138,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8305","reported_seq":8534,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.400779+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:20.400779+0000","last_peered":"2024-04-26T02:31:20.400779+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:20.400779+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26966278,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1394}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8466","reported_seq":8683,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.856295+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:19.856295+0000","last_peered":"2024-04-26T02:31:19.856295+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:19.856295+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8466,"log_dups_size":0,"ondisk_log_size":8466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138310,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8655","reported_seq":8902,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.504539+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:20.504539+0000","last_peered":"2024-04-26T02:31:20.504539+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:20.504539+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28398092,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8450","reported_seq":12660,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:19.004046+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:19.004046+0000","last_peered":"2024-04-26T02:31:19.004046+0000","last_clean":"2024-04-26T02:31:19.004046+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:19.004046+0000","last_undegraded":"2024-04-26T02:31:19.004046+0000","last_fullsized":"2024-04-26T02:31:19.004046+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:22.514565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890188,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8429","reported_seq":8658,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:18.041240+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:18.041240+0000","last_peered":"2024-04-26T02:31:18.041240+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:18.041240+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27587084,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":438131904,"num_objects":53490,"num_object_clones":0,"num_object_copies":213960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134067,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":877252608,"data_stored":877252608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134094,"ondisk_log_size":134094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674600,"kb_used_data":213920,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93697240,"statfs":{"total":96636764160,"available":95945973760,"internally_reserved":0,"allocated":219054080,"data_stored":218850418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,1,1,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683016,"kb_used_data":214528,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93688824,"statfs":{"total":96636764160,"available":95937355776,"internally_reserved":0,"allocated":219676672,"data_stored":219473010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571992,"kb_used_data":215504,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93799848,"statfs":{"total":96636764160,"available":96051044352,"internally_reserved":0,"allocated":220676096,"data_stored":220472434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680888,"kb_used_data":216108,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93690952,"statfs":{"total":96636764160,"available":95939534848,"internally_reserved":0,"allocated":221294592,"data_stored":221090930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220495872,"data_stored":220495872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219774976,"data_stored":219774976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218763264,"data_stored":218763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":218218496,"data_stored":218218496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:22.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:24.703 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:24.704 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:24.773 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-04-26T02:31:24.639024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434748608,"num_objects":53077,"num_object_clones":0,"num_object_copies":212308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134480,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134507,"ondisk_log_size":134507,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2610104,"kb_used_data":851352,"kb_used_omap":14,"kb_used_meta":1758705,"kb_avail":374877256,"statfs":{"total":386547056640,"available":383874310144,"internally_reserved":0,"allocated":871784448,"data_stored":870969800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1800914483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,3,9,10,8,15,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21766144,"num_objects":-2657,"num_object_clones":0,"num_object_copies":-10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015666"},"pg_stats":[{"pgid":"1.f","version":"38'8245","reported_seq":8477,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.547123+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:20.547123+0000","last_peered":"2024-04-26T02:31:20.547123+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:20.547123+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26861568,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8530","reported_seq":8757,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:23.042598+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:23.042598+0000","last_peered":"2024-04-26T02:31:23.042598+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:23.042598+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27046412,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8286","reported_seq":8455,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.493353+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:19.493353+0000","last_peered":"2024-04-26T02:31:19.493353+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:19.493353+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8286,"log_dups_size":0,"ondisk_log_size":8286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27392286,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8285,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1361}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8243","reported_seq":12268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:24.056372+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:24.056372+0000","last_peered":"2024-04-26T02:31:24.056372+0000","last_clean":"2024-04-26T02:31:24.056372+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:24.056372+0000","last_undegraded":"2024-04-26T02:31:24.056372+0000","last_fullsized":"2024-04-26T02:31:24.056372+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8243,"log_dups_size":0,"ondisk_log_size":8243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:26.383579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26513932,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8298","reported_seq":8527,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.467036+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:20.467036+0000","last_peered":"2024-04-26T02:31:20.467036+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:20.467036+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021836,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8325","reported_seq":8581,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.444569+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:20.444569+0000","last_peered":"2024-04-26T02:31:20.444569+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:20.444569+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26828800,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8325,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8474","reported_seq":12725,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:23.118375+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:23.118375+0000","last_peered":"2024-04-26T02:31:23.118375+0000","last_clean":"2024-04-26T02:31:23.118375+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:23.118375+0000","last_undegraded":"2024-04-26T02:31:23.118375+0000","last_fullsized":"2024-04-26T02:31:23.118375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:24.578307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26898956,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8472,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8691","reported_seq":8945,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:22.652919+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:22.652919+0000","last_peered":"2024-04-26T02:31:22.652919+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:22.652919+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27013644,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8362","reported_seq":9546,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.492343+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:20.492343+0000","last_peered":"2024-04-26T02:31:20.492343+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:20.492343+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27665432,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8288","reported_seq":8461,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.958351+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:19.958351+0000","last_peered":"2024-04-26T02:31:19.958351+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:19.958351+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8288,"log_dups_size":0,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26833420,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8331","reported_seq":8564,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.504662+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:20.504662+0000","last_peered":"2024-04-26T02:31:20.504662+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:20.504662+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27749138,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8305","reported_seq":8534,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.400779+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:20.400779+0000","last_peered":"2024-04-26T02:31:20.400779+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:20.400779+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8305,"log_dups_size":0,"ondisk_log_size":8305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26966278,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1394}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8466","reported_seq":8683,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:19.856295+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:19.856295+0000","last_peered":"2024-04-26T02:31:19.856295+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:19.856295+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8466,"log_dups_size":0,"ondisk_log_size":8466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138310,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8655","reported_seq":8902,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:20.504539+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:20.504539+0000","last_peered":"2024-04-26T02:31:20.504539+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:20.504539+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28398092,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8512","reported_seq":12722,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:24.091701+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:24.091701+0000","last_peered":"2024-04-26T02:31:24.091701+0000","last_clean":"2024-04-26T02:31:24.091701+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:24.091701+0000","last_undegraded":"2024-04-26T02:31:24.091701+0000","last_fullsized":"2024-04-26T02:31:24.091701+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8512,"log_dups_size":0,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:28.328240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27382284,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8510,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8496","reported_seq":8725,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:23.076752+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:23.076752+0000","last_peered":"2024-04-26T02:31:23.076752+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:23.076752+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038220,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":434748608,"num_objects":53077,"num_object_clones":0,"num_object_copies":212308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134480,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":868085760,"data_stored":868085760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134507,"ondisk_log_size":134507,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674600,"kb_used_data":213920,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93697240,"statfs":{"total":96636764160,"available":95945973760,"internally_reserved":0,"allocated":219054080,"data_stored":218850418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,1,1,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683016,"kb_used_data":214528,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93688824,"statfs":{"total":96636764160,"available":95937355776,"internally_reserved":0,"allocated":219676672,"data_stored":219473010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571872,"kb_used_data":211224,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93799968,"statfs":{"total":96636764160,"available":96051167232,"internally_reserved":0,"allocated":216293376,"data_stored":216089714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680616,"kb_used_data":211680,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93691224,"statfs":{"total":96636764160,"available":95939813376,"internally_reserved":0,"allocated":216760320,"data_stored":216556658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,4,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215928832,"data_stored":215928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215175168,"data_stored":215175168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218763264,"data_stored":218763264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":218218496,"data_stored":218218496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:24.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:24.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8245", 'reported_seq': 8477, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.547123+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:31:20.547123+0000', 'last_peered': '2024-04-26T02:31:20.547123+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:31:20.547123+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8245, 'log_dups_size': 0, 'ondisk_log_size': 8245, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26861568, 'num_objects': 3279, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8245, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1352}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8530", 'reported_seq': 8757, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:23.042598+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:31:23.042598+0000', 'last_peered': '2024-04-26T02:31:23.042598+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:31:23.042598+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8530, 'log_dups_size': 0, 'ondisk_log_size': 8530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27046412, 'num_objects': 3302, 'num_object_clones': 0, 'num_object_copies': 13208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8528, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8286", 'reported_seq': 8455, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:19.493353+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:31:19.493353+0000', 'last_peered': '2024-04-26T02:31:19.493353+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:31:19.493353+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8286, 'log_dups_size': 0, 'ondisk_log_size': 8286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27392286, 'num_objects': 3345, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3345, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8285, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1361}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8298", 'reported_seq': 8527, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.467036+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:20.467036+0000', 'last_peered': '2024-04-26T02:31:20.467036+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:20.467036+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8298, 'log_dups_size': 0, 'ondisk_log_size': 8298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27021836, 'num_objects': 3299, 'num_object_clones': 0, 'num_object_copies': 13196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8296, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1370}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8325", 'reported_seq': 8581, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.444569+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:31:20.444569+0000', 'last_peered': '2024-04-26T02:31:20.444569+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:31:20.444569+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8325, 'log_dups_size': 0, 'ondisk_log_size': 8325, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26828800, 'num_objects': 3275, 'num_object_clones': 0, 'num_object_copies': 13100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8325, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1325}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8691", 'reported_seq': 8945, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:22.652919+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:31:22.652919+0000', 'last_peered': '2024-04-26T02:31:22.652919+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:31:22.652919+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8691, 'log_dups_size': 0, 'ondisk_log_size': 8691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27013644, 'num_objects': 3298, 'num_object_clones': 0, 'num_object_copies': 13192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8689, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8362", 'reported_seq': 9546, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.492343+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:20.492343+0000', 'last_peered': '2024-04-26T02:31:20.492343+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:20.492343+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8362, 'log_dups_size': 0, 'ondisk_log_size': 8362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27665432, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 13512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1449, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8359, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 561, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8288", 'reported_seq': 8461, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:19.958351+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:31:19.958351+0000', 'last_peered': '2024-04-26T02:31:19.958351+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:31:19.958351+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8288, 'log_dups_size': 0, 'ondisk_log_size': 8288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26833420, 'num_objects': 3276, 'num_object_clones': 0, 'num_object_copies': 13104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8286, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1281}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8331", 'reported_seq': 8564, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.504662+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:31:20.504662+0000', 'last_peered': '2024-04-26T02:31:20.504662+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:31:20.504662+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8331, 'log_dups_size': 0, 'ondisk_log_size': 8331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27749138, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 13552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8328, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:24.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8305", 'reported_seq': 8534, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.400779+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:31:20.400779+0000', 'last_peered': '2024-04-26T02:31:20.400779+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:31:20.400779+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8305, 'log_dups_size': 0, 'ondisk_log_size': 8305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26966278, 'num_objects': 3292, 'num_object_clones': 0, 'num_object_copies': 13168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8304, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1394}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8466", 'reported_seq': 8683, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:19.856295+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:31:19.856295+0000', 'last_peered': '2024-04-26T02:31:19.856295+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:31:19.856295+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8466, 'log_dups_size': 0, 'ondisk_log_size': 8466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27138310, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8465, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'8655", 'reported_seq': 8902, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:20.504539+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:20.504539+0000', 'last_peered': '2024-04-26T02:31:20.504539+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:20.504539+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8655, 'log_dups_size': 0, 'ondisk_log_size': 8655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28398092, 'num_objects': 3467, 'num_object_clones': 0, 'num_object_copies': 13868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8653, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:24.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8496", 'reported_seq': 8725, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:23.076752+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:31:23.076752+0000', 'last_peered': '2024-04-26T02:31:23.076752+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:31:23.076752+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8496, 'log_dups_size': 0, 'ondisk_log_size': 8496, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27038220, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8494, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:24.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:26.878 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:26.878 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25147/212308 objects degraded (11.845%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":483,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53077,"data_bytes":434748608,"bytes_used":2672746496,"bytes_avail":383874310144,"bytes_total":386547056640,"degraded_objects":25147,"degraded_total":212308,"degraded_ratio":0.11844584283211183,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:26.898 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:26.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:28.322 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:28.322 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:28.479 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-04-26T02:31:26.639526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428981440,"num_objects":52373,"num_object_clones":0,"num_object_copies":209492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135184,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7666,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135211,"ondisk_log_size":135211,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2609636,"kb_used_data":842560,"kb_used_omap":14,"kb_used_meta":1767025,"kb_avail":374877724,"statfs":{"total":386547056640,"available":383874789376,"internally_reserved":0,"allocated":862781440,"data_stored":861966792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1809434163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,10,11,8,14,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23994368,"num_objects":-2929,"num_object_clones":0,"num_object_copies":-11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2929,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012526"},"pg_stats":[{"pgid":"1.f","version":"38'8323","reported_seq":8554,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.683047+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:25.683047+0000","last_peered":"2024-04-26T02:31:25.683047+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:25.683047+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26222592,"num_objects":3201,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8530","reported_seq":8757,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:23.042598+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:23.042598+0000","last_peered":"2024-04-26T02:31:23.042598+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:23.042598+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27046412,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8347","reported_seq":8516,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:24.934382+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:24.934382+0000","last_peered":"2024-04-26T02:31:24.934382+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:24.934382+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26892574,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8346,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8243","reported_seq":12268,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:24.056372+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:24.056372+0000","last_peered":"2024-04-26T02:31:24.056372+0000","last_clean":"2024-04-26T02:31:24.056372+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:24.056372+0000","last_undegraded":"2024-04-26T02:31:24.056372+0000","last_fullsized":"2024-04-26T02:31:24.056372+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8243,"log_dups_size":0,"ondisk_log_size":8243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:26.383579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26513932,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8362","reported_seq":8592,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.440431+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:25.440431+0000","last_peered":"2024-04-26T02:31:25.440431+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:25.440431+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26497548,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8392","reported_seq":8648,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.296232+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:25.296232+0000","last_peered":"2024-04-26T02:31:25.296232+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:25.296232+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8474","reported_seq":12725,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:23.118375+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:23.118375+0000","last_peered":"2024-04-26T02:31:23.118375+0000","last_clean":"2024-04-26T02:31:23.118375+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:23.118375+0000","last_undegraded":"2024-04-26T02:31:23.118375+0000","last_fullsized":"2024-04-26T02:31:23.118375+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:24.578307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26898956,"num_objects":3284,"num_object_clones":0,"num_object_copies":13136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8472,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8691","reported_seq":8945,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:22.652919+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:22.652919+0000","last_peered":"2024-04-26T02:31:22.652919+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:22.652919+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27013644,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8429","reported_seq":9701,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.495150+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:25.495150+0000","last_peered":"2024-04-26T02:31:25.495150+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:25.495150+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116568,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8426,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8372","reported_seq":8545,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.014390+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:25.014390+0000","last_peered":"2024-04-26T02:31:25.014390+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:25.014390+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8372,"log_dups_size":0,"ondisk_log_size":8372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26145292,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8370,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1197}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8409","reported_seq":8643,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.684998+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:25.684998+0000","last_peered":"2024-04-26T02:31:25.684998+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:25.684998+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27110162,"num_objects":3310,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8406,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8369","reported_seq":8597,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.625317+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:25.625317+0000","last_peered":"2024-04-26T02:31:25.625317+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:25.625317+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26441990,"num_objects":3228,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8368,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1330}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8534","reported_seq":8751,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:24.961666+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:24.961666+0000","last_peered":"2024-04-26T02:31:24.961666+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:24.961666+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26581254,"num_objects":3245,"num_object_clones":0,"num_object_copies":12980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8533,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8728","reported_seq":8975,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:25.495451+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:25.495451+0000","last_peered":"2024-04-26T02:31:25.495451+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:25.495451+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27800076,"num_objects":3394,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1415}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8512","reported_seq":12722,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:24.091701+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:24.091701+0000","last_peered":"2024-04-26T02:31:24.091701+0000","last_clean":"2024-04-26T02:31:24.091701+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:24.091701+0000","last_undegraded":"2024-04-26T02:31:24.091701+0000","last_fullsized":"2024-04-26T02:31:24.091701+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8512,"log_dups_size":0,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:28.328240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27382284,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8510,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8496","reported_seq":8725,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:23.076752+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:23.076752+0000","last_peered":"2024-04-26T02:31:23.076752+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:23.076752+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27038220,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428981440,"num_objects":52373,"num_object_clones":0,"num_object_copies":209492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135184,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7666,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":858836992,"data_stored":858836992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135211,"ondisk_log_size":135211,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674360,"kb_used_data":209520,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93697480,"statfs":{"total":96636764160,"available":95946219520,"internally_reserved":0,"allocated":214548480,"data_stored":214344818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682788,"kb_used_data":210136,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93689052,"statfs":{"total":96636764160,"available":95937589248,"internally_reserved":0,"allocated":215179264,"data_stored":214975602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571872,"kb_used_data":211224,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93799968,"statfs":{"total":96636764160,"available":96051167232,"internally_reserved":0,"allocated":216293376,"data_stored":216089714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680616,"kb_used_data":211680,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93691224,"statfs":{"total":96636764160,"available":95939813376,"internally_reserved":0,"allocated":216760320,"data_stored":216556658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,4,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215928832,"data_stored":215928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215175168,"data_stored":215175168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214159360,"data_stored":214159360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":213573632,"data_stored":213573632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:31.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:33.334 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:33.335 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:33.436 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-04-26T02:31:32.641047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420093120,"num_objects":51288,"num_object_clones":0,"num_object_copies":205152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136269,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7723,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136296,"ondisk_log_size":136296,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2616044,"kb_used_data":825352,"kb_used_omap":14,"kb_used_meta":1790641,"kb_avail":374871316,"statfs":{"total":386547056640,"available":383868227584,"internally_reserved":0,"allocated":845160448,"data_stored":844345800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1833616947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,8,16,18,22,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18038784,"num_objects":-2202,"num_object_clones":0,"num_object_copies":-8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005415"},"pg_stats":[{"pgid":"1.f","version":"38'8391","reported_seq":8623,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.511907+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:30.511907+0000","last_peered":"2024-04-26T02:31:30.511907+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:30.511907+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25665536,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8592","reported_seq":8820,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:27.936493+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:27.936493+0000","last_peered":"2024-04-26T02:31:27.936493+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:27.936493+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26538508,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8590,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8417","reported_seq":8586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.050695+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:30.050695+0000","last_peered":"2024-04-26T02:31:30.050695+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:30.050695+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8417,"log_dups_size":0,"ondisk_log_size":8417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319134,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8416,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8301","reported_seq":12326,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:29.022437+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:29.022437+0000","last_peered":"2024-04-26T02:31:29.022437+0000","last_clean":"2024-04-26T02:31:29.022437+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:29.022437+0000","last_undegraded":"2024-04-26T02:31:29.022437+0000","last_fullsized":"2024-04-26T02:31:29.022437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:32.402978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26038796,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8423","reported_seq":8653,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.737591+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:30.737591+0000","last_peered":"2024-04-26T02:31:30.737591+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:30.737591+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8423,"log_dups_size":0,"ondisk_log_size":8423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25997836,"num_objects":3174,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8459","reported_seq":8715,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.373554+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:30.373554+0000","last_peered":"2024-04-26T02:31:30.373554+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:30.373554+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25731072,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8536","reported_seq":12788,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:27.921394+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:27.921394+0000","last_peered":"2024-04-26T02:31:27.921394+0000","last_clean":"2024-04-26T02:31:27.921394+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:27.921394+0000","last_undegraded":"2024-04-26T02:31:27.921394+0000","last_fullsized":"2024-04-26T02:31:27.921394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:30.525016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391052,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8767","reported_seq":9021,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:27.875727+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:27.875727+0000","last_peered":"2024-04-26T02:31:27.875727+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:27.875727+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8767,"log_dups_size":0,"ondisk_log_size":8767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391052,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1170}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8511","reported_seq":9895,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.757635+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:30.757635+0000","last_peered":"2024-04-26T02:31:30.757635+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:30.757635+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26444824,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8450","reported_seq":8623,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.065077+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:30.065077+0000","last_peered":"2024-04-26T02:31:30.065077+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:30.065077+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25506316,"num_objects":3114,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8472","reported_seq":8705,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.784725+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:30.784725+0000","last_peered":"2024-04-26T02:31:30.784725+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:30.784725+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26594066,"num_objects":3247,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8419","reported_seq":8648,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.505433+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:30.505433+0000","last_peered":"2024-04-26T02:31:30.505433+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:30.505433+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26032390,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8418,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8600","reported_seq":8817,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:29.927338+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:29.927338+0000","last_peered":"2024-04-26T02:31:29.927338+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:29.927338+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26040582,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8816","reported_seq":9063,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.745426+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:30.745426+0000","last_peered":"2024-04-26T02:31:30.745426+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:30.745426+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079180,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1327}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8574","reported_seq":12784,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:29.023518+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:29.023518+0000","last_peered":"2024-04-26T02:31:29.023518+0000","last_clean":"2024-04-26T02:31:29.023518+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:29.023518+0000","last_undegraded":"2024-04-26T02:31:29.023518+0000","last_fullsized":"2024-04-26T02:31:29.023518+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:33.403403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26874380,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8568","reported_seq":8796,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:28.087657+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:28.087657+0000","last_peered":"2024-04-26T02:31:28.087657+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:28.087657+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8568,"log_dups_size":0,"ondisk_log_size":8568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26448396,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8566,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420093120,"num_objects":51288,"num_object_clones":0,"num_object_copies":205152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136269,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7723,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":840949760,"data_stored":840949760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136296,"ondisk_log_size":136296,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117083,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692376,"kb_used_data":205328,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93679464,"statfs":{"total":96636764160,"available":95927771136,"internally_reserved":0,"allocated":210255872,"data_stored":210052210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":498724560},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,10,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684020,"kb_used_data":205800,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93687820,"statfs":{"total":96636764160,"available":95936327680,"internally_reserved":0,"allocated":210739200,"data_stored":210535538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571644,"kb_used_data":206836,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93800196,"statfs":{"total":96636764160,"available":96051400704,"internally_reserved":0,"allocated":211800064,"data_stored":211596402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668004,"kb_used_data":207388,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93703836,"statfs":{"total":96636764160,"available":95952728064,"internally_reserved":0,"allocated":212365312,"data_stored":212161650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,4,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211492864,"data_stored":211492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210718720,"data_stored":210718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209625088,"data_stored":209625088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209113088,"data_stored":209113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:33.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:35.117 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:35.117 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:35.219 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-04-26T02:31:34.641557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416783552,"num_objects":50884,"num_object_clones":0,"num_object_copies":203536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136673,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7723,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136700,"ondisk_log_size":136700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2674576,"kb_used_data":815848,"kb_used_omap":14,"kb_used_meta":1858673,"kb_avail":374812784,"statfs":{"total":386547056640,"available":383808290816,"internally_reserved":0,"allocated":835428352,"data_stored":834613704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1903281715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,9,13,19,20,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21348352,"num_objects":-2606,"num_object_clones":0,"num_object_copies":-10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"1.f","version":"38'8391","reported_seq":8623,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.511907+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:30.511907+0000","last_peered":"2024-04-26T02:31:30.511907+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:30.511907+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25665536,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8662","reported_seq":8890,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:33.042661+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:33.042661+0000","last_peered":"2024-04-26T02:31:33.042661+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:33.042661+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25965068,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1170}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8417","reported_seq":8586,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.050695+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:30.050695+0000","last_peered":"2024-04-26T02:31:30.050695+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:30.050695+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8417,"log_dups_size":0,"ondisk_log_size":8417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26319134,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8416,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8362","reported_seq":12387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:34.200286+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:34.200286+0000","last_peered":"2024-04-26T02:31:34.200286+0000","last_clean":"2024-04-26T02:31:34.200286+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:34.200286+0000","last_undegraded":"2024-04-26T02:31:34.200286+0000","last_fullsized":"2024-04-26T02:31:34.200286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.345886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539084,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8423","reported_seq":8653,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.737591+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:30.737591+0000","last_peered":"2024-04-26T02:31:30.737591+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:30.737591+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8423,"log_dups_size":0,"ondisk_log_size":8423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25997836,"num_objects":3174,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8459","reported_seq":8715,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.373554+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:30.373554+0000","last_peered":"2024-04-26T02:31:30.373554+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:30.373554+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25731072,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8604","reported_seq":12856,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:33.019964+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:33.019964+0000","last_peered":"2024-04-26T02:31:33.019964+0000","last_clean":"2024-04-26T02:31:33.019964+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:33.019964+0000","last_undegraded":"2024-04-26T02:31:33.019964+0000","last_fullsized":"2024-04-26T02:31:33.019964+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8604,"log_dups_size":0,"ondisk_log_size":8604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:35.596442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25833996,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8602,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8847","reported_seq":9100,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:33.021927+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:33.021927+0000","last_peered":"2024-04-26T02:31:33.021927+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:33.021927+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25735692,"num_objects":3142,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8511","reported_seq":9895,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.757635+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:30.757635+0000","last_peered":"2024-04-26T02:31:30.757635+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:30.757635+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26444824,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8450","reported_seq":8623,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.065077+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:30.065077+0000","last_peered":"2024-04-26T02:31:30.065077+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:30.065077+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25506316,"num_objects":3114,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8472","reported_seq":8705,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.784725+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:30.784725+0000","last_peered":"2024-04-26T02:31:30.784725+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:30.784725+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26594066,"num_objects":3247,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8419","reported_seq":8648,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.505433+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:30.505433+0000","last_peered":"2024-04-26T02:31:30.505433+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:30.505433+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26032390,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8418,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1280}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8600","reported_seq":8817,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:29.927338+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:29.927338+0000","last_peered":"2024-04-26T02:31:29.927338+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:29.927338+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26040582,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8816","reported_seq":9063,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:30.745426+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:30.745426+0000","last_peered":"2024-04-26T02:31:30.745426+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:30.745426+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079180,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1327}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8633","reported_seq":12843,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:34.039496+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:34.039496+0000","last_peered":"2024-04-26T02:31:34.039496+0000","last_clean":"2024-04-26T02:31:34.039496+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:34.039496+0000","last_undegraded":"2024-04-26T02:31:34.039496+0000","last_fullsized":"2024-04-26T02:31:34.039496+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391052,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8634","reported_seq":8863,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:33.120744+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:33.120744+0000","last_peered":"2024-04-26T02:31:33.120744+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:33.120744+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8634,"log_dups_size":0,"ondisk_log_size":8634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25907724,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416783552,"num_objects":50884,"num_object_clones":0,"num_object_copies":203536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136673,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7723,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":831971328,"data_stored":831971328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136700,"ondisk_log_size":136700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117083,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692376,"kb_used_data":205328,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93679464,"statfs":{"total":96636764160,"available":95927771136,"internally_reserved":0,"allocated":210255872,"data_stored":210052210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":498724560},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,10,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684020,"kb_used_data":205800,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93687820,"statfs":{"total":96636764160,"available":95936327680,"internally_reserved":0,"allocated":210739200,"data_stored":210535538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612320,"kb_used_data":201688,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93759520,"statfs":{"total":96636764160,"available":96009748480,"internally_reserved":0,"allocated":206528512,"data_stored":206324850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685860,"kb_used_data":203032,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93685980,"statfs":{"total":96636764160,"available":95934443520,"internally_reserved":0,"allocated":207904768,"data_stored":207701106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494399183},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207024128,"data_stored":207024128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206209024,"data_stored":206209024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209625088,"data_stored":209625088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209113088,"data_stored":209113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:35.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8391", 'reported_seq': 8623, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.511907+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:31:30.511907+0000', 'last_peered': '2024-04-26T02:31:30.511907+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:31:30.511907+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8391, 'log_dups_size': 0, 'ondisk_log_size': 8391, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25665536, 'num_objects': 3133, 'num_object_clones': 0, 'num_object_copies': 12532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8391, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1206}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8662", 'reported_seq': 8890, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:33.042661+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:31:33.042661+0000', 'last_peered': '2024-04-26T02:31:33.042661+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:31:33.042661+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8662, 'log_dups_size': 0, 'ondisk_log_size': 8662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25965068, 'num_objects': 3170, 'num_object_clones': 0, 'num_object_copies': 12680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1170}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8417", 'reported_seq': 8586, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.050695+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:31:30.050695+0000', 'last_peered': '2024-04-26T02:31:30.050695+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:31:30.050695+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8417, 'log_dups_size': 0, 'ondisk_log_size': 8417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26319134, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8416, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8423", 'reported_seq': 8653, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.737591+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:30.737591+0000', 'last_peered': '2024-04-26T02:31:30.737591+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:30.737591+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8423, 'log_dups_size': 0, 'ondisk_log_size': 8423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25997836, 'num_objects': 3174, 'num_object_clones': 0, 'num_object_copies': 12696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8421, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.221 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8459", 'reported_seq': 8715, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.373554+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:31:30.373554+0000', 'last_peered': '2024-04-26T02:31:30.373554+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:31:30.373554+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8459, 'log_dups_size': 0, 'ondisk_log_size': 8459, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25731072, 'num_objects': 3141, 'num_object_clones': 0, 'num_object_copies': 12564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8459, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8847", 'reported_seq': 9100, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:33.021927+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:31:33.021927+0000', 'last_peered': '2024-04-26T02:31:33.021927+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:31:33.021927+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8847, 'log_dups_size': 0, 'ondisk_log_size': 8847, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25735692, 'num_objects': 3142, 'num_object_clones': 0, 'num_object_copies': 12568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8845, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8511", 'reported_seq': 9895, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.757635+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:30.757635+0000', 'last_peered': '2024-04-26T02:31:30.757635+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:30.757635+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8511, 'log_dups_size': 0, 'ondisk_log_size': 8511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26444824, 'num_objects': 3229, 'num_object_clones': 0, 'num_object_copies': 12916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1348, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8508, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 662, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8450", 'reported_seq': 8623, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.065077+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:31:30.065077+0000', 'last_peered': '2024-04-26T02:31:30.065077+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:31:30.065077+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8450, 'log_dups_size': 0, 'ondisk_log_size': 8450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25506316, 'num_objects': 3114, 'num_object_clones': 0, 'num_object_copies': 12456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8448, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1119}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:35.222 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8472", 'reported_seq': 8705, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.784725+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:31:30.784725+0000', 'last_peered': '2024-04-26T02:31:30.784725+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:31:30.784725+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8472, 'log_dups_size': 0, 'ondisk_log_size': 8472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26594066, 'num_objects': 3247, 'num_object_clones': 0, 'num_object_copies': 12988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8469, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1313}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8419", 'reported_seq': 8648, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.505433+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:31:30.505433+0000', 'last_peered': '2024-04-26T02:31:30.505433+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:31:30.505433+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8419, 'log_dups_size': 0, 'ondisk_log_size': 8419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26032390, 'num_objects': 3178, 'num_object_clones': 0, 'num_object_copies': 12712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8418, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8600", 'reported_seq': 8817, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:29.927338+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:31:29.927338+0000', 'last_peered': '2024-04-26T02:31:29.927338+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:31:29.927338+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8600, 'log_dups_size': 0, 'ondisk_log_size': 8600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26040582, 'num_objects': 3179, 'num_object_clones': 0, 'num_object_copies': 12716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8599, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'8816", 'reported_seq': 9063, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:30.745426+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:30.745426+0000', 'last_peered': '2024-04-26T02:31:30.745426+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:30.745426+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8816, 'log_dups_size': 0, 'ondisk_log_size': 8816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27079180, 'num_objects': 3306, 'num_object_clones': 0, 'num_object_copies': 13224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8814, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:35.223 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8634", 'reported_seq': 8863, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:33.120744+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:31:33.120744+0000', 'last_peered': '2024-04-26T02:31:33.120744+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:31:33.120744+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8634, 'log_dups_size': 0, 'ondisk_log_size': 8634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25907724, 'num_objects': 3163, 'num_object_clones': 0, 'num_object_copies': 12652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8632, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:35.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:37.590 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:37.590 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25004/200816 objects degraded (12.451%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":494,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50204,"data_bytes":411212992,"bytes_used":2724806656,"bytes_avail":383822249984,"bytes_total":386547056640,"degraded_objects":25004,"degraded_total":200816,"degraded_ratio":0.12451199107640826,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":239},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T02:30:34.605252+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:37.679 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:37.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:39.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:39.988 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:40.095 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-04-26T02:31:38.642540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408739008,"num_objects":49902,"num_object_clones":0,"num_object_copies":199608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137655,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137682,"ondisk_log_size":137682,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660696,"kb_used_data":801968,"kb_used_omap":14,"kb_used_meta":1858673,"kb_avail":374826664,"statfs":{"total":386547056640,"available":383822503936,"internally_reserved":0,"allocated":821215232,"data_stored":820400584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1903281715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,7,11,11,11,10,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20242432,"num_objects":-2471,"num_object_clones":0,"num_object_copies":-9884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003013"},"pg_stats":[{"pgid":"1.f","version":"38'8448","reported_seq":8680,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.803032+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:35.803032+0000","last_peered":"2024-04-26T02:31:35.803032+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:35.803032+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8741","reported_seq":8969,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:37.951097+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:37.951097+0000","last_peered":"2024-04-26T02:31:37.951097+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:37.951097+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8741,"log_dups_size":0,"ondisk_log_size":8741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25317900,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1091}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8495","reported_seq":8663,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.138441+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:35.138441+0000","last_peered":"2024-04-26T02:31:35.138441+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:35.138441+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8495,"log_dups_size":0,"ondisk_log_size":8495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25680158,"num_objects":3136,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8494,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8362","reported_seq":12387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:34.200286+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:34.200286+0000","last_peered":"2024-04-26T02:31:34.200286+0000","last_clean":"2024-04-26T02:31:34.200286+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:34.200286+0000","last_undegraded":"2024-04-26T02:31:34.200286+0000","last_fullsized":"2024-04-26T02:31:34.200286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:38.345886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25539084,"num_objects":3118,"num_object_clones":0,"num_object_copies":12472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8490","reported_seq":8720,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.511595+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:35.511595+0000","last_peered":"2024-04-26T02:31:35.511595+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:35.511595+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25448972,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8536","reported_seq":8792,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.765876+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:35.765876+0000","last_peered":"2024-04-26T02:31:35.765876+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:35.765876+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25100288,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8686","reported_seq":12938,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:38.003052+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:38.003052+0000","last_peered":"2024-04-26T02:31:38.003052+0000","last_clean":"2024-04-26T02:31:38.003052+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:38.003052+0000","last_undegraded":"2024-04-26T02:31:38.003052+0000","last_fullsized":"2024-04-26T02:31:38.003052+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:41.615024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25162252,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8913","reported_seq":9167,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:38.152946+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:38.152946+0000","last_peered":"2024-04-26T02:31:38.152946+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:38.152946+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25195020,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8578","reported_seq":10047,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.841842+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:35.841842+0000","last_peered":"2024-04-26T02:31:35.841842+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:35.841842+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25895960,"num_objects":3162,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8575,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8513","reported_seq":8686,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.137527+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:35.137527+0000","last_peered":"2024-04-26T02:31:35.137527+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:35.137527+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24990220,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8533","reported_seq":8767,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.811726+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:35.811726+0000","last_peered":"2024-04-26T02:31:35.811726+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:35.811726+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8533,"log_dups_size":0,"ondisk_log_size":8533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26094354,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8499","reported_seq":8728,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.917640+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:35.917640+0000","last_peered":"2024-04-26T02:31:35.917640+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:35.917640+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8499,"log_dups_size":0,"ondisk_log_size":8499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25377030,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8666","reported_seq":8883,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.061180+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:35.061180+0000","last_peered":"2024-04-26T02:31:35.061180+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:35.061180+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8666,"log_dups_size":0,"ondisk_log_size":8666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25499910,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8665,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1068}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8880","reported_seq":9127,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:35.581432+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:35.581432+0000","last_peered":"2024-04-26T02:31:35.581432+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:35.581432+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554892,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8633","reported_seq":12843,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:34.039496+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:34.039496+0000","last_peered":"2024-04-26T02:31:34.039496+0000","last_clean":"2024-04-26T02:31:34.039496+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:34.039496+0000","last_undegraded":"2024-04-26T02:31:34.039496+0000","last_fullsized":"2024-04-26T02:31:34.039496+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391052,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8709","reported_seq":8938,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:37.978396+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:37.978396+0000","last_peered":"2024-04-26T02:31:37.978396+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:37.978396+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293324,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":408739008,"num_objects":49902,"num_object_clones":0,"num_object_copies":199608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137655,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818544640,"data_stored":818544640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137682,"ondisk_log_size":137682,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678984,"kb_used_data":200248,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93692856,"statfs":{"total":96636764160,"available":95941484544,"internally_reserved":0,"allocated":205053952,"data_stored":204850290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":490204880},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,3,0,2,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683780,"kb_used_data":201404,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93688060,"statfs":{"total":96636764160,"available":95936573440,"internally_reserved":0,"allocated":206237696,"data_stored":206034034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612320,"kb_used_data":201688,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93759520,"statfs":{"total":96636764160,"available":96009748480,"internally_reserved":0,"allocated":206528512,"data_stored":206324850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685612,"kb_used_data":198628,"kb_used_omap":4,"kb_used_meta":486971,"kb_avail":93686228,"statfs":{"total":96636764160,"available":95934697472,"internally_reserved":0,"allocated":203395072,"data_stored":203191410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498659023},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,6,2,3,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202575872,"data_stored":202575872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206209024,"data_stored":206209024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205176832,"data_stored":205176832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204582912,"data_stored":204582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:43.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:44.952 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:44.954 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:45.434 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-04-26T02:31:44.647062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398941376,"num_objects":48706,"num_object_clones":0,"num_object_copies":194824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138851,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7810,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138878,"ondisk_log_size":138878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722136,"kb_used_data":780144,"kb_used_omap":14,"kb_used_meta":1941937,"kb_avail":374765224,"statfs":{"total":386547056640,"available":383759589376,"internally_reserved":0,"allocated":798867456,"data_stored":798052808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1988544051},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,10,9,16,15,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21151744,"num_objects":-2582,"num_object_clones":0,"num_object_copies":-10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2582,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006014"},"pg_stats":[{"pgid":"1.f","version":"38'8520","reported_seq":8752,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.712473+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:40.712473+0000","last_peered":"2024-04-26T02:31:40.712473+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:40.712473+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8520,"log_dups_size":0,"ondisk_log_size":8520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24608768,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8788","reported_seq":9016,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.090921+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:43.090921+0000","last_peered":"2024-04-26T02:31:43.090921+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:43.090921+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24932876,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8786,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8547","reported_seq":8715,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.164294+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:40.164294+0000","last_peered":"2024-04-26T02:31:40.164294+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:40.164294+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25254174,"num_objects":3084,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8546,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8490","reported_seq":12515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.385646+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:44.385646+0000","last_peered":"2024-04-26T02:31:44.385646+0000","last_clean":"2024-04-26T02:31:44.385646+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:44.385646+0000","last_undegraded":"2024-04-26T02:31:44.385646+0000","last_fullsized":"2024-04-26T02:31:44.385646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:49.404164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24490508,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8563","reported_seq":8793,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.804799+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:40.804799+0000","last_peered":"2024-04-26T02:31:40.804799+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:40.804799+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8563,"log_dups_size":0,"ondisk_log_size":8563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24850956,"num_objects":3034,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8561,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8603","reported_seq":8859,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:41.055461+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:41.055461+0000","last_peered":"2024-04-26T02:31:41.055461+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:41.055461+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8603,"log_dups_size":0,"ondisk_log_size":8603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8754","reported_seq":13006,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:43.008478+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:43.008478+0000","last_peered":"2024-04-26T02:31:43.008478+0000","last_clean":"2024-04-26T02:31:43.008478+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:43.008478+0000","last_undegraded":"2024-04-26T02:31:43.008478+0000","last_fullsized":"2024-04-26T02:31:43.008478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:46.744446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24605196,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8752,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8971","reported_seq":9225,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.087550+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:43.087550+0000","last_peered":"2024-04-26T02:31:43.087550+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:43.087550+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8971,"log_dups_size":0,"ondisk_log_size":8971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24719884,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8645","reported_seq":10205,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.947952+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:40.947952+0000","last_peered":"2024-04-26T02:31:40.947952+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:40.947952+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8645,"log_dups_size":0,"ondisk_log_size":8645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25347096,"num_objects":3095,"num_object_clones":0,"num_object_copies":12380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":749,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8579","reported_seq":8752,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.242881+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:40.242881+0000","last_peered":"2024-04-26T02:31:40.242881+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:40.242881+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24449548,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":990}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8604","reported_seq":8838,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.976326+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:40.976326+0000","last_peered":"2024-04-26T02:31:40.976326+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:40.976326+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8604,"log_dups_size":0,"ondisk_log_size":8604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25512722,"num_objects":3115,"num_object_clones":0,"num_object_copies":12460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8575","reported_seq":8804,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.860329+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:40.860329+0000","last_peered":"2024-04-26T02:31:40.860329+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:40.860329+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8575,"log_dups_size":0,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24754438,"num_objects":3022,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8574,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8727","reported_seq":8943,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.347915+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:40.347915+0000","last_peered":"2024-04-26T02:31:40.347915+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:40.347915+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8727,"log_dups_size":0,"ondisk_log_size":8727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25000198,"num_objects":3052,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'8966","reported_seq":9211,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:40.984351+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:40.984351+0000","last_peered":"2024-04-26T02:31:40.984351+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:40.984351+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8966,"log_dups_size":0,"ondisk_log_size":8966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25850380,"num_objects":3156,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1177}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8768","reported_seq":12978,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.371801+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:44.371801+0000","last_peered":"2024-04-26T02:31:44.371801+0000","last_clean":"2024-04-26T02:31:44.371801+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:44.371801+0000","last_undegraded":"2024-04-26T02:31:44.371801+0000","last_fullsized":"2024-04-26T02:31:44.371801+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:45.326371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285132,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8778","reported_seq":9007,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.088219+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:43.088219+0000","last_peered":"2024-04-26T02:31:43.088219+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:43.088219+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24728076,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8776,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":398941376,"num_objects":48706,"num_object_clones":0,"num_object_copies":194824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138851,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7810,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":796143616,"data_stored":796143616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138878,"ondisk_log_size":138878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696840,"kb_used_data":195900,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93675000,"statfs":{"total":96636764160,"available":95923200000,"internally_reserved":0,"allocated":200601600,"data_stored":200397938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705776,"kb_used_data":197032,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93666064,"statfs":{"total":96636764160,"available":95914049536,"internally_reserved":0,"allocated":201760768,"data_stored":201557106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,11,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616100,"kb_used_data":192984,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93755740,"statfs":{"total":96636764160,"available":96005877760,"internally_reserved":0,"allocated":197615616,"data_stored":197411954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":433254095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703420,"kb_used_data":194228,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93668420,"statfs":{"total":96636764160,"available":95916462080,"internally_reserved":0,"allocated":198889472,"data_stored":198685810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198144000,"data_stored":198144000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197189632,"data_stored":197189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200753152,"data_stored":200753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200056832,"data_stored":200056832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:45.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:47.123 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:47.124 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:47.140 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-04-26T02:31:46.647602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":393206976,"num_objects":48006,"num_object_clones":0,"num_object_copies":192024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139551,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7850,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139578,"ondisk_log_size":139578,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705116,"kb_used_data":771440,"kb_used_omap":14,"kb_used_meta":1933617,"kb_avail":374782244,"statfs":{"total":386547056640,"available":383777017856,"internally_reserved":0,"allocated":789954560,"data_stored":789139912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1980024371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,10,8,10,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23576576,"num_objects":-2878,"num_object_clones":0,"num_object_copies":-11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2878,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006044"},"pg_stats":[{"pgid":"1.f","version":"38'8573","reported_seq":8805,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.063374+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:46.063374+0000","last_peered":"2024-04-26T02:31:46.063374+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:46.063374+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24174592,"num_objects":2951,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8788","reported_seq":9016,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.090921+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:43.090921+0000","last_peered":"2024-04-26T02:31:43.090921+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:43.090921+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24932876,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8786,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8623","reported_seq":8792,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.216259+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:45.216259+0000","last_peered":"2024-04-26T02:31:45.216259+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:45.216259+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24631582,"num_objects":3008,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8622,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8490","reported_seq":12515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.385646+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:44.385646+0000","last_peered":"2024-04-26T02:31:44.385646+0000","last_clean":"2024-04-26T02:31:44.385646+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:44.385646+0000","last_undegraded":"2024-04-26T02:31:44.385646+0000","last_fullsized":"2024-04-26T02:31:44.385646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:49.404164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24490508,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8622","reported_seq":8852,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.978122+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:45.978122+0000","last_peered":"2024-04-26T02:31:45.978122+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:45.978122+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24367628,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8689","reported_seq":8945,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.028149+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:46.028149+0000","last_peered":"2024-04-26T02:31:46.028149+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:46.028149+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23846912,"num_objects":2911,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8754","reported_seq":13006,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:43.008478+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:43.008478+0000","last_peered":"2024-04-26T02:31:43.008478+0000","last_clean":"2024-04-26T02:31:43.008478+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:43.008478+0000","last_undegraded":"2024-04-26T02:31:43.008478+0000","last_fullsized":"2024-04-26T02:31:43.008478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:46.744446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24605196,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8752,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'8971","reported_seq":9225,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.087550+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:43.087550+0000","last_peered":"2024-04-26T02:31:43.087550+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:43.087550+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8971,"log_dups_size":0,"ondisk_log_size":8971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24719884,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8707","reported_seq":10346,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.090552+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:46.090552+0000","last_peered":"2024-04-26T02:31:46.090552+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:46.090552+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24839192,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8704,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8662","reported_seq":8834,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.252924+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:45.252924+0000","last_peered":"2024-04-26T02:31:45.252924+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:45.252924+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23769612,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8661","reported_seq":8895,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.022809+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:46.022809+0000","last_peered":"2024-04-26T02:31:46.022809+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:46.022809+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25045778,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8641","reported_seq":8870,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.670800+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:45.670800+0000","last_peered":"2024-04-26T02:31:45.670800+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:45.670800+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8641,"log_dups_size":0,"ondisk_log_size":8641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24213766,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8640,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8814","reported_seq":9029,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.323030+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:45.323030+0000","last_peered":"2024-04-26T02:31:45.323030+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:45.323030+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287494,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9037","reported_seq":9284,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.979710+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:45.979710+0000","last_peered":"2024-04-26T02:31:45.979710+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:45.979710+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25268748,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8768","reported_seq":12978,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.371801+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:44.371801+0000","last_peered":"2024-04-26T02:31:44.371801+0000","last_clean":"2024-04-26T02:31:44.371801+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:44.371801+0000","last_undegraded":"2024-04-26T02:31:44.371801+0000","last_fullsized":"2024-04-26T02:31:44.371801+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:45.326371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285132,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8778","reported_seq":9007,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:43.088219+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:43.088219+0000","last_peered":"2024-04-26T02:31:43.088219+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:43.088219+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24728076,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8776,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":393206976,"num_objects":48006,"num_object_clones":0,"num_object_copies":192024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139551,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7850,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":787189760,"data_stored":787189760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139578,"ondisk_log_size":139578,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696684,"kb_used_data":191580,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93675156,"statfs":{"total":96636764160,"available":95923359744,"internally_reserved":0,"allocated":196177920,"data_stored":195974258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688912,"kb_used_data":192648,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93682928,"statfs":{"total":96636764160,"available":95931318272,"internally_reserved":0,"allocated":197271552,"data_stored":197067890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616100,"kb_used_data":192984,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93755740,"statfs":{"total":96636764160,"available":96005877760,"internally_reserved":0,"allocated":197615616,"data_stored":197411954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":433254095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703420,"kb_used_data":194228,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93668420,"statfs":{"total":96636764160,"available":95916462080,"internally_reserved":0,"allocated":198889472,"data_stored":198685810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198144000,"data_stored":198144000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197189632,"data_stored":197189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196251648,"data_stored":196251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195604480,"data_stored":195604480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8573", 'reported_seq': 8805, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:46.063374+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:31:46.063374+0000', 'last_peered': '2024-04-26T02:31:46.063374+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:31:46.063374+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8573, 'log_dups_size': 0, 'ondisk_log_size': 8573, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24174592, 'num_objects': 2951, 'num_object_clones': 0, 'num_object_copies': 11804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8573, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1024}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8788", 'reported_seq': 9016, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:43.090921+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:31:43.090921+0000', 'last_peered': '2024-04-26T02:31:43.090921+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:31:43.090921+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8788, 'log_dups_size': 0, 'ondisk_log_size': 8788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24932876, 'num_objects': 3044, 'num_object_clones': 0, 'num_object_copies': 12176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8786, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8623", 'reported_seq': 8792, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.216259+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:31:45.216259+0000', 'last_peered': '2024-04-26T02:31:45.216259+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:31:45.216259+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8623, 'log_dups_size': 0, 'ondisk_log_size': 8623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24631582, 'num_objects': 3008, 'num_object_clones': 0, 'num_object_copies': 12032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3008, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8622, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1024}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:47.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8622", 'reported_seq': 8852, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.978122+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:45.978122+0000', 'last_peered': '2024-04-26T02:31:45.978122+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:45.978122+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8622, 'log_dups_size': 0, 'ondisk_log_size': 8622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24367628, 'num_objects': 2975, 'num_object_clones': 0, 'num_object_copies': 11900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8620, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1046}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8689", 'reported_seq': 8945, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:46.028149+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:31:46.028149+0000', 'last_peered': '2024-04-26T02:31:46.028149+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:31:46.028149+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8689, 'log_dups_size': 0, 'ondisk_log_size': 8689, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23846912, 'num_objects': 2911, 'num_object_clones': 0, 'num_object_copies': 11644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8689, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'8971", 'reported_seq': 9225, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:43.087550+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:31:43.087550+0000', 'last_peered': '2024-04-26T02:31:43.087550+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:31:43.087550+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8971, 'log_dups_size': 0, 'ondisk_log_size': 8971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24719884, 'num_objects': 3018, 'num_object_clones': 0, 'num_object_copies': 12072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8969, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8707", 'reported_seq': 10346, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:46.090552+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:46.090552+0000', 'last_peered': '2024-04-26T02:31:46.090552+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:46.090552+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8707, 'log_dups_size': 0, 'ondisk_log_size': 8707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24839192, 'num_objects': 3033, 'num_object_clones': 0, 'num_object_copies': 12132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8704, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 789, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1812}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8662", 'reported_seq': 8834, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.252924+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:31:45.252924+0000', 'last_peered': '2024-04-26T02:31:45.252924+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:31:45.252924+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8662, 'log_dups_size': 0, 'ondisk_log_size': 8662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23769612, 'num_objects': 2902, 'num_object_clones': 0, 'num_object_copies': 11608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8661", 'reported_seq': 8895, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:46.022809+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:31:46.022809+0000', 'last_peered': '2024-04-26T02:31:46.022809+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:31:46.022809+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8661, 'log_dups_size': 0, 'ondisk_log_size': 8661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25045778, 'num_objects': 3058, 'num_object_clones': 0, 'num_object_copies': 12232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8658, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8641", 'reported_seq': 8870, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.670800+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:31:45.670800+0000', 'last_peered': '2024-04-26T02:31:45.670800+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:31:45.670800+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8641, 'log_dups_size': 0, 'ondisk_log_size': 8641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24213766, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 11824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8640, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8814", 'reported_seq': 9029, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.323030+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:31:45.323030+0000', 'last_peered': '2024-04-26T02:31:45.323030+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:31:45.323030+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8814, 'log_dups_size': 0, 'ondisk_log_size': 8814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24287494, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8813, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9037", 'reported_seq': 9284, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:45.979710+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:45.979710+0000', 'last_peered': '2024-04-26T02:31:45.979710+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:45.979710+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9037, 'log_dups_size': 0, 'ondisk_log_size': 9037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25268748, 'num_objects': 3085, 'num_object_clones': 0, 'num_object_copies': 12340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9035, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:47.143 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:47.144 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8778", 'reported_seq': 9007, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:43.088219+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:31:43.088219+0000', 'last_peered': '2024-04-26T02:31:43.088219+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:31:43.088219+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8778, 'log_dups_size': 0, 'ondisk_log_size': 8778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24728076, 'num_objects': 3019, 'num_object_clones': 0, 'num_object_copies': 12076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8776, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:47.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:48.794 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24919/190920 objects degraded (13.052%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":505,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47730,"data_bytes":390945984,"bytes_used":2757156864,"bytes_avail":383789899776,"bytes_total":386547056640,"degraded_objects":24919,"degraded_total":190920,"degraded_ratio":0.13052063691598575,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:48.873 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:48.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:50.147 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:50.148 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:50.187 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-04-26T02:31:48.648131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":390945984,"num_objects":47730,"num_object_clones":0,"num_object_copies":190920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139827,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7850,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139854,"ondisk_log_size":139854,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2692536,"kb_used_data":767180,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374794824,"statfs":{"total":386547056640,"available":383789899776,"internally_reserved":0,"allocated":785592320,"data_stored":784777672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1971504691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,7,7,11,9,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20267008,"num_objects":-2474,"num_object_clones":0,"num_object_copies":-9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006069"},"pg_stats":[{"pgid":"1.f","version":"38'8573","reported_seq":8805,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.063374+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:46.063374+0000","last_peered":"2024-04-26T02:31:46.063374+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:46.063374+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24174592,"num_objects":2951,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8853","reported_seq":9081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:48.263126+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:48.263126+0000","last_peered":"2024-04-26T02:31:48.263126+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:48.263126+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8853,"log_dups_size":0,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24400396,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":979}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8623","reported_seq":8792,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.216259+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:45.216259+0000","last_peered":"2024-04-26T02:31:45.216259+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:45.216259+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24631582,"num_objects":3008,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8622,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8490","reported_seq":12515,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.385646+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:44.385646+0000","last_peered":"2024-04-26T02:31:44.385646+0000","last_clean":"2024-04-26T02:31:44.385646+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:44.385646+0000","last_undegraded":"2024-04-26T02:31:44.385646+0000","last_fullsized":"2024-04-26T02:31:44.385646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:49.404164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24490508,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8622","reported_seq":8852,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.978122+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:45.978122+0000","last_peered":"2024-04-26T02:31:45.978122+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:45.978122+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24367628,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8689","reported_seq":8945,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.028149+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:46.028149+0000","last_peered":"2024-04-26T02:31:46.028149+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:46.028149+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23846912,"num_objects":2911,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8812","reported_seq":13064,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:48.147246+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:48.147246+0000","last_peered":"2024-04-26T02:31:48.147246+0000","last_clean":"2024-04-26T02:31:48.147246+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:48.147246+0000","last_undegraded":"2024-04-26T02:31:48.147246+0000","last_fullsized":"2024-04-26T02:31:48.147246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:51.763267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24130060,"num_objects":2946,"num_object_clones":0,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9045","reported_seq":9298,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:48.219820+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:48.219820+0000","last_peered":"2024-04-26T02:31:48.219820+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:48.219820+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24113676,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8707","reported_seq":10346,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.090552+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:46.090552+0000","last_peered":"2024-04-26T02:31:46.090552+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:46.090552+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24839192,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8704,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8662","reported_seq":8834,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.252924+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:45.252924+0000","last_peered":"2024-04-26T02:31:45.252924+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:45.252924+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23769612,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8661","reported_seq":8895,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:46.022809+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:46.022809+0000","last_peered":"2024-04-26T02:31:46.022809+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:46.022809+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25045778,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8641","reported_seq":8870,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.670800+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:45.670800+0000","last_peered":"2024-04-26T02:31:45.670800+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:45.670800+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8641,"log_dups_size":0,"ondisk_log_size":8641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24213766,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8640,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8814","reported_seq":9029,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.323030+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:45.323030+0000","last_peered":"2024-04-26T02:31:45.323030+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:45.323030+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287494,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9037","reported_seq":9284,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:45.979710+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:45.979710+0000","last_peered":"2024-04-26T02:31:45.979710+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:45.979710+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25268748,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8768","reported_seq":12978,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:44.371801+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:44.371801+0000","last_peered":"2024-04-26T02:31:44.371801+0000","last_clean":"2024-04-26T02:31:44.371801+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:44.371801+0000","last_undegraded":"2024-04-26T02:31:44.371801+0000","last_fullsized":"2024-04-26T02:31:44.371801+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:45.326371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25285132,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8857","reported_seq":9080,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:48.199134+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:48.199134+0000","last_peered":"2024-04-26T02:31:48.199134+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:48.199134+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24080908,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8855,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":390945984,"num_objects":47730,"num_object_clones":0,"num_object_copies":190920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139827,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7850,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":782897152,"data_stored":782897152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139854,"ondisk_log_size":139854,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696684,"kb_used_data":191580,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93675156,"statfs":{"total":96636764160,"available":95923359744,"internally_reserved":0,"allocated":196177920,"data_stored":195974258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,0,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688912,"kb_used_data":192648,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93682928,"statfs":{"total":96636764160,"available":95931318272,"internally_reserved":0,"allocated":197271552,"data_stored":197067890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616100,"kb_used_data":192984,"kb_used_omap":4,"kb_used_meta":423099,"kb_avail":93755740,"statfs":{"total":96636764160,"available":96005877760,"internally_reserved":0,"allocated":197615616,"data_stored":197411954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":433254095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690840,"kb_used_data":189968,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93681000,"statfs":{"total":96636764160,"available":95929344000,"internally_reserved":0,"allocated":194527232,"data_stored":194323570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,0,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193851392,"data_stored":193851392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197189632,"data_stored":197189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196251648,"data_stored":196251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195604480,"data_stored":195604480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:53.189 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:55.188 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:55.189 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:55.488 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-26T02:31:54.649690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":382401728,"num_objects":46687,"num_object_clones":0,"num_object_copies":186748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46687,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140870,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140897,"ondisk_log_size":140897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2675832,"kb_used_data":750480,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374811528,"statfs":{"total":386547056640,"available":383807004672,"internally_reserved":0,"allocated":768491520,"data_stored":767676872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1971504691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,11,15,13,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19578880,"num_objects":-2390,"num_object_clones":0,"num_object_copies":-9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003150"},"pg_stats":[{"pgid":"1.f","version":"38'8618","reported_seq":8850,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.842834+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:50.842834+0000","last_peered":"2024-04-26T02:31:50.842834+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:50.842834+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8930","reported_seq":9158,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:52.973643+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:52.973643+0000","last_peered":"2024-04-26T02:31:52.973643+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:52.973643+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23769612,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8689","reported_seq":8856,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.368406+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:50.368406+0000","last_peered":"2024-04-26T02:31:50.368406+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:50.368406+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24090910,"num_objects":2942,"num_object_clones":0,"num_object_copies":11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8688,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8557","reported_seq":12582,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:49.522888+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:49.522888+0000","last_peered":"2024-04-26T02:31:49.522888+0000","last_clean":"2024-04-26T02:31:49.522888+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:49.522888+0000","last_undegraded":"2024-04-26T02:31:49.522888+0000","last_fullsized":"2024-04-26T02:31:49.522888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8557,"log_dups_size":0,"ondisk_log_size":8557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23941644,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8691","reported_seq":8921,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.998082+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:50.998082+0000","last_peered":"2024-04-26T02:31:50.998082+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:50.998082+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23802380,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8758","reported_seq":9014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:51.084441+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:51.084441+0000","last_peered":"2024-04-26T02:31:51.084441+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:51.084441+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8758,"log_dups_size":0,"ondisk_log_size":8758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23281664,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8758,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8871","reported_seq":13122,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:53.225724+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:53.225724+0000","last_peered":"2024-04-26T02:31:53.225724+0000","last_clean":"2024-04-26T02:31:53.225724+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:53.225724+0000","last_undegraded":"2024-04-26T02:31:53.225724+0000","last_fullsized":"2024-04-26T02:31:53.225724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8871,"log_dups_size":0,"ondisk_log_size":8871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:56.945642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23646732,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9106","reported_seq":9360,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:53.003739+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:53.003739+0000","last_peered":"2024-04-26T02:31:53.003739+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:53.003739+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23613964,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8773","reported_seq":10499,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:51.119162+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:51.119162+0000","last_peered":"2024-04-26T02:31:51.119162+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:51.119162+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24298520,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":832,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8731","reported_seq":8904,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.218872+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:50.218872+0000","last_peered":"2024-04-26T02:31:50.218872+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:50.218872+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8731,"log_dups_size":0,"ondisk_log_size":8731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23204364,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8729","reported_seq":8963,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.998076+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:50.998076+0000","last_peered":"2024-04-26T02:31:50.998076+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:50.998076+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24488722,"num_objects":2990,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8716","reported_seq":8945,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:51.113573+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:51.113573+0000","last_peered":"2024-04-26T02:31:51.113573+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:51.113573+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23599366,"num_objects":2881,"num_object_clones":0,"num_object_copies":11524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":983}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8870","reported_seq":9086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.376646+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:50.376646+0000","last_peered":"2024-04-26T02:31:50.376646+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:50.376646+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8870,"log_dups_size":0,"ondisk_log_size":8870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23828742,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9106","reported_seq":9353,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:50.852952+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:50.852952+0000","last_peered":"2024-04-26T02:31:50.852952+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:50.852952+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24703500,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8830","reported_seq":13040,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:49.481922+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:49.481922+0000","last_peered":"2024-04-26T02:31:49.481922+0000","last_clean":"2024-04-26T02:31:49.481922+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:49.481922+0000","last_undegraded":"2024-04-26T02:31:49.481922+0000","last_fullsized":"2024-04-26T02:31:49.481922+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:50.560956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777228,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8922","reported_seq":9150,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:53.044036+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:53.044036+0000","last_peered":"2024-04-26T02:31:53.044036+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:53.044036+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23548428,"num_objects":2875,"num_object_clones":0,"num_object_copies":11500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":382401728,"num_objects":46687,"num_object_clones":0,"num_object_copies":186748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46687,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140870,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":765837312,"data_stored":765837312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140897,"ondisk_log_size":140897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117087,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696760,"kb_used_data":187500,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93675080,"statfs":{"total":96636764160,"available":95923281920,"internally_reserved":0,"allocated":192000000,"data_stored":191796338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,6,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689028,"kb_used_data":188604,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93682812,"statfs":{"total":96636764160,"available":95931199488,"internally_reserved":0,"allocated":193130496,"data_stored":192926834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599380,"kb_used_data":188748,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93772460,"statfs":{"total":96636764160,"available":96022999040,"internally_reserved":0,"allocated":193277952,"data_stored":193074290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690664,"kb_used_data":185628,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93681176,"statfs":{"total":96636764160,"available":95929524224,"internally_reserved":0,"allocated":190083072,"data_stored":189879410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189530112,"data_stored":189530112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192917504,"data_stored":192917504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192032768,"data_stored":192032768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":191356928,"data_stored":191356928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:55.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:31:56.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:56.883 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:31:56.900 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-04-26T02:31:56.650193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376036544,"num_objects":45910,"num_object_clones":0,"num_object_copies":183640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141647,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7950,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141674,"ondisk_log_size":141674,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2663232,"kb_used_data":737872,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374824128,"statfs":{"total":386547056640,"available":383819907072,"internally_reserved":0,"allocated":755580928,"data_stored":754766280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1971504691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,8,8,10,20,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22904832,"num_objects":-2796,"num_object_clones":0,"num_object_copies":-11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003130"},"pg_stats":[{"pgid":"1.f","version":"38'8681","reported_seq":8910,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.827433+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:55.827433+0000","last_peered":"2024-04-26T02:31:55.827433+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:55.827433+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23289856,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8681,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8930","reported_seq":9158,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:52.973643+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:52.973643+0000","last_peered":"2024-04-26T02:31:52.973643+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:52.973643+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23769612,"num_objects":2902,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8753","reported_seq":8922,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.346553+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:55.346553+0000","last_peered":"2024-04-26T02:31:55.346553+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:55.346553+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23566622,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8752,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8615","reported_seq":12640,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:54.622072+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:54.622072+0000","last_peered":"2024-04-26T02:31:54.622072+0000","last_clean":"2024-04-26T02:31:54.622072+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:54.622072+0000","last_undegraded":"2024-04-26T02:31:54.622072+0000","last_fullsized":"2024-04-26T02:31:54.622072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23466508,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8613,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8753","reported_seq":8983,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.922704+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:55.922704+0000","last_peered":"2024-04-26T02:31:55.922704+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:55.922704+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23294476,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8815","reported_seq":9071,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.101392+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:56.101392+0000","last_peered":"2024-04-26T02:31:56.101392+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:56.101392+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22814720,"num_objects":2785,"num_object_clones":0,"num_object_copies":11140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8871","reported_seq":13122,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:53.225724+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:53.225724+0000","last_peered":"2024-04-26T02:31:53.225724+0000","last_clean":"2024-04-26T02:31:53.225724+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:53.225724+0000","last_undegraded":"2024-04-26T02:31:53.225724+0000","last_fullsized":"2024-04-26T02:31:53.225724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8871,"log_dups_size":0,"ondisk_log_size":8871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:56.945642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23646732,"num_objects":2887,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9106","reported_seq":9360,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:53.003739+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:53.003739+0000","last_peered":"2024-04-26T02:31:53.003739+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:53.003739+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23613964,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8849","reported_seq":10689,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.040711+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:56.040711+0000","last_peered":"2024-04-26T02:31:56.040711+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:56.040711+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23675928,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8793","reported_seq":8966,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.370106+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:55.370106+0000","last_peered":"2024-04-26T02:31:55.370106+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:55.370106+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22696460,"num_objects":2771,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8815","reported_seq":9047,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.168710+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:56.168710+0000","last_peered":"2024-04-26T02:31:56.168710+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:56.168710+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23784210,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8764","reported_seq":8993,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.071407+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:56.071407+0000","last_peered":"2024-04-26T02:31:56.071407+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:56.071407+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23206150,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8763,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8937","reported_seq":9154,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.253529+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:55.253529+0000","last_peered":"2024-04-26T02:31:55.253529+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:55.253529+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279878,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9174","reported_seq":9421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.127478+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:56.127478+0000","last_peered":"2024-04-26T02:31:56.127478+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:56.127478+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24146444,"num_objects":2948,"num_object_clones":0,"num_object_copies":11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8896","reported_seq":13106,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:54.561035+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:54.561035+0000","last_peered":"2024-04-26T02:31:54.561035+0000","last_clean":"2024-04-26T02:31:54.561035+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:54.561035+0000","last_undegraded":"2024-04-26T02:31:54.561035+0000","last_fullsized":"2024-04-26T02:31:54.561035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:55.621564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24236556,"num_objects":2959,"num_object_clones":0,"num_object_copies":11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8922","reported_seq":9150,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:53.044036+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:53.044036+0000","last_peered":"2024-04-26T02:31:53.044036+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:53.044036+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23548428,"num_objects":2875,"num_object_clones":0,"num_object_copies":11500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":888}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376036544,"num_objects":45910,"num_object_clones":0,"num_object_copies":183640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141647,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7950,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":752812032,"data_stored":752812032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141674,"ondisk_log_size":141674,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684288,"kb_used_data":183352,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93687552,"statfs":{"total":96636764160,"available":95936053248,"internally_reserved":0,"allocated":187752448,"data_stored":187548786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,0,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688956,"kb_used_data":184360,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93682884,"statfs":{"total":96636764160,"available":95931273216,"internally_reserved":0,"allocated":188784640,"data_stored":188580978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599324,"kb_used_data":184532,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93772516,"statfs":{"total":96636764160,"available":96023056384,"internally_reserved":0,"allocated":188960768,"data_stored":188757106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690664,"kb_used_data":185628,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93681176,"statfs":{"total":96636764160,"available":95929524224,"internally_reserved":0,"allocated":190083072,"data_stored":189879410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,1,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189530112,"data_stored":189530112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188567552,"data_stored":188567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187686912,"data_stored":187686912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187027456,"data_stored":187027456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8681", 'reported_seq': 8910, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:55.827433+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:31:55.827433+0000', 'last_peered': '2024-04-26T02:31:55.827433+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:31:55.827433+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8681, 'log_dups_size': 0, 'ondisk_log_size': 8681, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23289856, 'num_objects': 2843, 'num_object_clones': 0, 'num_object_copies': 11372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8681, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 916}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'8930", 'reported_seq': 9158, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:52.973643+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:31:52.973643+0000', 'last_peered': '2024-04-26T02:31:52.973643+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:31:52.973643+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8930, 'log_dups_size': 0, 'ondisk_log_size': 8930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23769612, 'num_objects': 2902, 'num_object_clones': 0, 'num_object_copies': 11608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8928, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:31:56.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8753", 'reported_seq': 8922, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:55.346553+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:31:55.346553+0000', 'last_peered': '2024-04-26T02:31:55.346553+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:31:55.346553+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8753, 'log_dups_size': 0, 'ondisk_log_size': 8753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23566622, 'num_objects': 2878, 'num_object_clones': 0, 'num_object_copies': 11512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2878, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8752, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 894}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8753", 'reported_seq': 8983, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:55.922704+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:31:55.922704+0000', 'last_peered': '2024-04-26T02:31:55.922704+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:31:55.922704+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8753, 'log_dups_size': 0, 'ondisk_log_size': 8753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23294476, 'num_objects': 2844, 'num_object_clones': 0, 'num_object_copies': 11376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8751, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 915}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8815", 'reported_seq': 9071, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:56.101392+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:31:56.101392+0000', 'last_peered': '2024-04-26T02:31:56.101392+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:31:56.101392+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8815, 'log_dups_size': 0, 'ondisk_log_size': 8815, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22814720, 'num_objects': 2785, 'num_object_clones': 0, 'num_object_copies': 11140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8815, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 835}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9106", 'reported_seq': 9360, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:53.003739+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:31:53.003739+0000', 'last_peered': '2024-04-26T02:31:53.003739+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:31:53.003739+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9106, 'log_dups_size': 0, 'ondisk_log_size': 9106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23613964, 'num_objects': 2883, 'num_object_clones': 0, 'num_object_copies': 11532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9104, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8849", 'reported_seq': 10689, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:56.040711+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:31:56.040711+0000', 'last_peered': '2024-04-26T02:31:56.040711+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:31:56.040711+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8849, 'log_dups_size': 0, 'ondisk_log_size': 8849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23675928, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8846, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 889, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:31:56.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8793", 'reported_seq': 8966, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:55.370106+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:31:55.370106+0000', 'last_peered': '2024-04-26T02:31:55.370106+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:31:55.370106+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8793, 'log_dups_size': 0, 'ondisk_log_size': 8793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22696460, 'num_objects': 2771, 'num_object_clones': 0, 'num_object_copies': 11084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8791, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 776}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8815", 'reported_seq': 9047, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:56.168710+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:31:56.168710+0000', 'last_peered': '2024-04-26T02:31:56.168710+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:31:56.168710+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8815, 'log_dups_size': 0, 'ondisk_log_size': 8815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23784210, 'num_objects': 2904, 'num_object_clones': 0, 'num_object_copies': 11616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8812, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 970}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8764", 'reported_seq': 8993, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:56.071407+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:31:56.071407+0000', 'last_peered': '2024-04-26T02:31:56.071407+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:31:56.071407+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8764, 'log_dups_size': 0, 'ondisk_log_size': 8764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23206150, 'num_objects': 2833, 'num_object_clones': 0, 'num_object_copies': 11332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8763, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 935}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'8937", 'reported_seq': 9154, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:55.253529+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:31:55.253529+0000', 'last_peered': '2024-04-26T02:31:55.253529+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:31:55.253529+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8937, 'log_dups_size': 0, 'ondisk_log_size': 8937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23279878, 'num_objects': 2842, 'num_object_clones': 0, 'num_object_copies': 11368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8936, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9174", 'reported_seq': 9421, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:56.127478+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:31:56.127478+0000', 'last_peered': '2024-04-26T02:31:56.127478+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:31:56.127478+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9174, 'log_dups_size': 0, 'ondisk_log_size': 9174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24146444, 'num_objects': 2948, 'num_object_clones': 0, 'num_object_copies': 11792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9172, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:31:56.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'8922", 'reported_seq': 9150, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:31:53.044036+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:31:53.044036+0000', 'last_peered': '2024-04-26T02:31:53.044036+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:31:53.044036+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8922, 'log_dups_size': 0, 'ondisk_log_size': 8922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23548428, 'num_objects': 2875, 'num_object_clones': 0, 'num_object_copies': 11500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8920, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:31:56.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:31:58.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:31:58.883 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24819/182616 objects degraded (13.591%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":515,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45654,"data_bytes":373939392,"bytes_used":2727198720,"bytes_avail":383819857920,"bytes_total":386547056640,"degraded_objects":24819,"degraded_total":182616,"degraded_ratio":0.13590813510316729,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:31:58.979 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:31:58.979 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:00.244 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:00.245 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:00.361 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-26T02:31:58.651379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373939392,"num_objects":45654,"num_object_clones":0,"num_object_copies":182616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141903,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7950,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141930,"ondisk_log_size":141930,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2663280,"kb_used_data":733764,"kb_used_omap":14,"kb_used_meta":1929457,"kb_avail":374824080,"statfs":{"total":386547056640,"available":383819857920,"internally_reserved":0,"allocated":751374336,"data_stored":750559688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1975764531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,9,4,10,19,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003777"},"pg_stats":[{"pgid":"1.f","version":"38'8681","reported_seq":8910,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.827433+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:31:55.827433+0000","last_peered":"2024-04-26T02:31:55.827433+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:31:55.827433+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23289856,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8681,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'8992","reported_seq":9220,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:58.228719+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:31:58.228719+0000","last_peered":"2024-04-26T02:31:58.228719+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:31:58.228719+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23261708,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":840}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8753","reported_seq":8922,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.346553+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:31:55.346553+0000","last_peered":"2024-04-26T02:31:55.346553+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:31:55.346553+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23566622,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8752,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":894}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8615","reported_seq":12640,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:54.622072+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:54.622072+0000","last_peered":"2024-04-26T02:31:54.622072+0000","last_clean":"2024-04-26T02:31:54.622072+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:54.622072+0000","last_undegraded":"2024-04-26T02:31:54.622072+0000","last_fullsized":"2024-04-26T02:31:54.622072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23466508,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8613,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8753","reported_seq":8983,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.922704+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:31:55.922704+0000","last_peered":"2024-04-26T02:31:55.922704+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:31:55.922704+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23294476,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8815","reported_seq":9071,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.101392+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:31:56.101392+0000","last_peered":"2024-04-26T02:31:56.101392+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:31:56.101392+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22814720,"num_objects":2785,"num_object_clones":0,"num_object_copies":11140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'8940","reported_seq":13190,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:58.171442+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:31:58.171442+0000","last_peered":"2024-04-26T02:31:58.171442+0000","last_clean":"2024-04-26T02:31:58.171442+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:31:58.171442+0000","last_undegraded":"2024-04-26T02:31:58.171442+0000","last_fullsized":"2024-04-26T02:31:58.171442+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:01.984361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23081484,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8938,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9166","reported_seq":9420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:58.190055+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:31:58.190055+0000","last_peered":"2024-04-26T02:31:58.190055+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:31:58.190055+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23122444,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9164,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":771}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8849","reported_seq":10689,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.040711+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:31:56.040711+0000","last_peered":"2024-04-26T02:31:56.040711+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:31:56.040711+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23675928,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8793","reported_seq":8966,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.370106+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:31:55.370106+0000","last_peered":"2024-04-26T02:31:55.370106+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:31:55.370106+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22696460,"num_objects":2771,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8815","reported_seq":9047,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.168710+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:31:56.168710+0000","last_peered":"2024-04-26T02:31:56.168710+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:31:56.168710+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23784210,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8764","reported_seq":8993,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.071407+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:31:56.071407+0000","last_peered":"2024-04-26T02:31:56.071407+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:31:56.071407+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23206150,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8763,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'8937","reported_seq":9154,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:55.253529+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:31:55.253529+0000","last_peered":"2024-04-26T02:31:55.253529+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:31:55.253529+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279878,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9174","reported_seq":9421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:56.127478+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:31:56.127478+0000","last_peered":"2024-04-26T02:31:56.127478+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:31:56.127478+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24146444,"num_objects":2948,"num_object_clones":0,"num_object_copies":11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8896","reported_seq":13106,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:54.561035+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:54.561035+0000","last_peered":"2024-04-26T02:31:54.561035+0000","last_clean":"2024-04-26T02:31:54.561035+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:54.561035+0000","last_undegraded":"2024-04-26T02:31:54.561035+0000","last_fullsized":"2024-04-26T02:31:54.561035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:55.621564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24236556,"num_objects":2959,"num_object_clones":0,"num_object_copies":11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'8987","reported_seq":9216,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:31:58.037890+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:31:58.037890+0000","last_peered":"2024-04-26T02:31:58.037890+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:31:58.037890+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015948,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":373939392,"num_objects":45654,"num_object_clones":0,"num_object_copies":182616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141903,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7950,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":748613632,"data_stored":748613632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141930,"ondisk_log_size":141930,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117088,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684288,"kb_used_data":183352,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93687552,"statfs":{"total":96636764160,"available":95936053248,"internally_reserved":0,"allocated":187752448,"data_stored":187548786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,0,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688956,"kb_used_data":184360,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93682884,"statfs":{"total":96636764160,"available":95931273216,"internally_reserved":0,"allocated":188784640,"data_stored":188580978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599324,"kb_used_data":184532,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93772516,"statfs":{"total":96636764160,"available":96023056384,"internally_reserved":0,"allocated":188960768,"data_stored":188757106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690712,"kb_used_data":181520,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93681128,"statfs":{"total":96636764160,"available":95929475072,"internally_reserved":0,"allocated":185876480,"data_stored":185672818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,0,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185331712,"data_stored":185331712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188567552,"data_stored":188567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187686912,"data_stored":187686912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187027456,"data_stored":187027456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:03.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:04.951 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:05.149 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-04-26T02:32:04.659325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365255872,"num_objects":44594,"num_object_clones":0,"num_object_copies":178376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44594,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142963,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8014,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142990,"ondisk_log_size":142990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2646780,"kb_used_data":717252,"kb_used_omap":14,"kb_used_meta":1929457,"kb_avail":374840580,"statfs":{"total":386547056640,"available":383836753920,"internally_reserved":0,"allocated":734466048,"data_stored":733651400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1975764531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,7,9,11,8,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19292160,"num_objects":-2355,"num_object_clones":0,"num_object_copies":-9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010149"},"pg_stats":[{"pgid":"1.f","version":"38'8738","reported_seq":8969,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.287750+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:01.287750+0000","last_peered":"2024-04-26T02:32:01.287750+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:01.287750+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22822912,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":859}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9068","reported_seq":9295,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.306183+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:03.306183+0000","last_peered":"2024-04-26T02:32:03.306183+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:03.306183+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22639116,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8797","reported_seq":8966,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:00.110503+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:00.110503+0000","last_peered":"2024-04-26T02:32:00.110503+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:00.110503+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23206174,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8796,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":850}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8662","reported_seq":12687,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:59.299404+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:31:59.299404+0000","last_peered":"2024-04-26T02:31:59.299404+0000","last_clean":"2024-04-26T02:31:59.299404+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:31:59.299404+0000","last_undegraded":"2024-04-26T02:31:59.299404+0000","last_fullsized":"2024-04-26T02:31:59.299404+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:00.592982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23081484,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8814","reported_seq":9044,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.074795+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:01.074795+0000","last_peered":"2024-04-26T02:32:01.074795+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:01.074795+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22794764,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8895","reported_seq":9151,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.244460+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:01.244460+0000","last_peered":"2024-04-26T02:32:01.244460+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:01.244460+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8895,"log_dups_size":0,"ondisk_log_size":8895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9012","reported_seq":13264,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:03.291931+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:03.291931+0000","last_peered":"2024-04-26T02:32:03.291931+0000","last_clean":"2024-04-26T02:32:03.291931+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:03.291931+0000","last_undegraded":"2024-04-26T02:32:03.291931+0000","last_fullsized":"2024-04-26T02:32:03.291931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:07.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22491660,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9236","reported_seq":9490,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.255107+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:03.255107+0000","last_peered":"2024-04-26T02:32:03.255107+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:03.255107+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549004,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8913","reported_seq":10881,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.246724+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:01.246724+0000","last_peered":"2024-04-26T02:32:01.246724+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:01.246724+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23151640,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8848","reported_seq":9021,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:00.352601+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:00.352601+0000","last_peered":"2024-04-26T02:32:00.352601+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:00.352601+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8848,"log_dups_size":0,"ondisk_log_size":8848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22245900,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":721}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8880","reported_seq":9114,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.063302+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:01.063302+0000","last_peered":"2024-04-26T02:32:01.063302+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:01.063302+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23251730,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8836","reported_seq":9060,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.096133+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:01.096133+0000","last_peered":"2024-04-26T02:32:01.096133+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:01.096133+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22616326,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":863}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9007","reported_seq":9224,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:00.234622+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:00.234622+0000","last_peered":"2024-04-26T02:32:00.234622+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:00.234622+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9007,"log_dups_size":0,"ondisk_log_size":9007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22706438,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9006,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9245","reported_seq":9492,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:01.177397+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:01.177397+0000","last_peered":"2024-04-26T02:32:01.177397+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:01.177397+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9245,"log_dups_size":0,"ondisk_log_size":9245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23564812,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9243,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":898}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'8983","reported_seq":13193,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:31:59.718745+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:31:59.718745+0000","last_peered":"2024-04-26T02:31:59.718745+0000","last_clean":"2024-04-26T02:31:59.718745+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:31:59.718745+0000","last_undegraded":"2024-04-26T02:31:59.718745+0000","last_fullsized":"2024-04-26T02:31:59.718745+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8983,"log_dups_size":0,"ondisk_log_size":8983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:01.637108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23523852,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9056","reported_seq":9285,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.164716+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:03.164716+0000","last_peered":"2024-04-26T02:32:03.164716+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:03.164716+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22450700,"num_objects":2741,"num_object_clones":0,"num_object_copies":10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365255872,"num_objects":44594,"num_object_clones":0,"num_object_copies":178376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44594,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142963,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8014,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":731549696,"data_stored":731549696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142990,"ondisk_log_size":142990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117089,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684300,"kb_used_data":179204,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93687540,"statfs":{"total":96636764160,"available":95936040960,"internally_reserved":0,"allocated":183504896,"data_stored":183301234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,7,2,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689072,"kb_used_data":180308,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93682768,"statfs":{"total":96636764160,"available":95931154432,"internally_reserved":0,"allocated":184635392,"data_stored":184431730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599400,"kb_used_data":180448,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93772440,"statfs":{"total":96636764160,"available":96022978560,"internally_reserved":0,"allocated":184778752,"data_stored":184575090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674008,"kb_used_data":177292,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93697832,"statfs":{"total":96636764160,"available":95946579968,"internally_reserved":0,"allocated":181547008,"data_stored":181343346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180957184,"data_stored":180957184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184266752,"data_stored":184266752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":183554048,"data_stored":183554048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":182771712,"data_stored":182771712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:05.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:06.823 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:06.824 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:06.901 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-26T02:32:06.659833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358751424,"num_objects":43800,"num_object_clones":0,"num_object_copies":175200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143757,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143784,"ondisk_log_size":143784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2620732,"kb_used_data":703704,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374866628,"statfs":{"total":386547056640,"available":383863427072,"internally_reserved":0,"allocated":720592896,"data_stored":719778248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1962985011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,10,16,8,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23650304,"num_objects":-2887,"num_object_clones":0,"num_object_copies":-11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010142"},"pg_stats":[{"pgid":"1.f","version":"38'8802","reported_seq":9034,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.290222+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:06.290222+0000","last_peered":"2024-04-26T02:32:06.290222+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:06.290222+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22298624,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9068","reported_seq":9295,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.306183+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:03.306183+0000","last_peered":"2024-04-26T02:32:03.306183+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:03.306183+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22639116,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8857","reported_seq":9026,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:05.190719+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:05.190719+0000","last_peered":"2024-04-26T02:32:05.190719+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:05.190719+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22714654,"num_objects":2774,"num_object_clones":0,"num_object_copies":11096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8856,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8718","reported_seq":12742,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:04.855788+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:04.855788+0000","last_peered":"2024-04-26T02:32:04.855788+0000","last_clean":"2024-04-26T02:32:04.855788+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:04.855788+0000","last_undegraded":"2024-04-26T02:32:04.855788+0000","last_fullsized":"2024-04-26T02:32:04.855788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8718,"log_dups_size":0,"ondisk_log_size":8718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:05.596791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22622732,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8716,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8872","reported_seq":9102,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.184620+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:06.184620+0000","last_peered":"2024-04-26T02:32:06.184620+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:06.184620+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8872,"log_dups_size":0,"ondisk_log_size":8872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22319628,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8870,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8964","reported_seq":9220,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.283770+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:06.283770+0000","last_peered":"2024-04-26T02:32:06.283770+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:06.283770+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21594112,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9012","reported_seq":13264,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:03.291931+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:03.291931+0000","last_peered":"2024-04-26T02:32:03.291931+0000","last_clean":"2024-04-26T02:32:03.291931+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:03.291931+0000","last_undegraded":"2024-04-26T02:32:03.291931+0000","last_fullsized":"2024-04-26T02:32:03.291931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9012,"log_dups_size":0,"ondisk_log_size":9012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:07.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22491660,"num_objects":2746,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9236","reported_seq":9490,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.255107+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:03.255107+0000","last_peered":"2024-04-26T02:32:03.255107+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:03.255107+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22549004,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8977","reported_seq":11046,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.301542+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:06.301542+0000","last_peered":"2024-04-26T02:32:06.301542+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:06.301542+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22627352,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8915","reported_seq":9087,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:05.459950+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:05.459950+0000","last_peered":"2024-04-26T02:32:05.459950+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:05.459950+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21697036,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8951","reported_seq":9183,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.396943+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:06.396943+0000","last_peered":"2024-04-26T02:32:06.396943+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:06.396943+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8951,"log_dups_size":0,"ondisk_log_size":8951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22670098,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8912","reported_seq":9141,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.221531+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:06.221531+0000","last_peered":"2024-04-26T02:32:06.221531+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:06.221531+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21993734,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9064","reported_seq":9281,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:05.334626+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:05.334626+0000","last_peered":"2024-04-26T02:32:05.334626+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:05.334626+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22239494,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":670}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9324","reported_seq":9571,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.138841+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:06.138841+0000","last_peered":"2024-04-26T02:32:06.138841+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:06.138841+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9324,"log_dups_size":0,"ondisk_log_size":9324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22917644,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9322,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9056","reported_seq":13266,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:04.646674+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:04.646674+0000","last_peered":"2024-04-26T02:32:04.646674+0000","last_clean":"2024-04-26T02:32:04.646674+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:04.646674+0000","last_undegraded":"2024-04-26T02:32:04.646674+0000","last_fullsized":"2024-04-26T02:32:04.646674+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:07.584332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22925836,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9056","reported_seq":9285,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:03.164716+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:03.164716+0000","last_peered":"2024-04-26T02:32:03.164716+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:03.164716+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22450700,"num_objects":2741,"num_object_clones":0,"num_object_copies":10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":358751424,"num_objects":43800,"num_object_clones":0,"num_object_copies":175200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143757,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":718196736,"data_stored":718196736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143784,"ondisk_log_size":143784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684168,"kb_used_data":174912,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93687672,"statfs":{"total":96636764160,"available":95936176128,"internally_reserved":0,"allocated":179109888,"data_stored":178906226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675688,"kb_used_data":175264,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93696152,"statfs":{"total":96636764160,"available":95944859648,"internally_reserved":0,"allocated":179470336,"data_stored":179266674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586868,"kb_used_data":176236,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93784972,"statfs":{"total":96636764160,"available":96035811328,"internally_reserved":0,"allocated":180465664,"data_stored":180262002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674008,"kb_used_data":177292,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93697832,"statfs":{"total":96636764160,"available":95946579968,"internally_reserved":0,"allocated":181547008,"data_stored":181343346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180957184,"data_stored":180957184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179818496,"data_stored":179818496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179105792,"data_stored":179105792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178315264,"data_stored":178315264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:06.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8802", 'reported_seq': 9034, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.290222+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:06.290222+0000', 'last_peered': '2024-04-26T02:32:06.290222+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:06.290222+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8802, 'log_dups_size': 0, 'ondisk_log_size': 8802, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22298624, 'num_objects': 2722, 'num_object_clones': 0, 'num_object_copies': 10888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8802, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 795}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9068", 'reported_seq': 9295, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:03.306183+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:03.306183+0000', 'last_peered': '2024-04-26T02:32:03.306183+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:03.306183+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9068, 'log_dups_size': 0, 'ondisk_log_size': 9068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22639116, 'num_objects': 2764, 'num_object_clones': 0, 'num_object_copies': 11056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9066, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8857", 'reported_seq': 9026, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:05.190719+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:05.190719+0000', 'last_peered': '2024-04-26T02:32:05.190719+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:05.190719+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8857, 'log_dups_size': 0, 'ondisk_log_size': 8857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22714654, 'num_objects': 2774, 'num_object_clones': 0, 'num_object_copies': 11096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2774, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8856, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 790}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8872", 'reported_seq': 9102, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.184620+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:06.184620+0000', 'last_peered': '2024-04-26T02:32:06.184620+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:06.184620+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8872, 'log_dups_size': 0, 'ondisk_log_size': 8872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22319628, 'num_objects': 2725, 'num_object_clones': 0, 'num_object_copies': 10900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8870, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 796}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'8964", 'reported_seq': 9220, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.283770+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:06.283770+0000', 'last_peered': '2024-04-26T02:32:06.283770+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:06.283770+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8964, 'log_dups_size': 0, 'ondisk_log_size': 8964, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21594112, 'num_objects': 2636, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8964, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 686}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:06.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9236", 'reported_seq': 9490, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:03.255107+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:03.255107+0000', 'last_peered': '2024-04-26T02:32:03.255107+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:03.255107+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9236, 'log_dups_size': 0, 'ondisk_log_size': 9236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22549004, 'num_objects': 2753, 'num_object_clones': 0, 'num_object_copies': 11012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9234, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'8977", 'reported_seq': 11046, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.301542+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:06.301542+0000', 'last_peered': '2024-04-26T02:32:06.301542+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:06.301542+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8977, 'log_dups_size': 0, 'ondisk_log_size': 8977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22627352, 'num_objects': 2763, 'num_object_clones': 0, 'num_object_copies': 11052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8974, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1004, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'8915", 'reported_seq': 9087, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:05.459950+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:05.459950+0000', 'last_peered': '2024-04-26T02:32:05.459950+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:05.459950+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8915, 'log_dups_size': 0, 'ondisk_log_size': 8915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21697036, 'num_objects': 2649, 'num_object_clones': 0, 'num_object_copies': 10596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8913, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 654}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'8951", 'reported_seq': 9183, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.396943+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:06.396943+0000', 'last_peered': '2024-04-26T02:32:06.396943+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:06.396943+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8951, 'log_dups_size': 0, 'ondisk_log_size': 8951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22670098, 'num_objects': 2768, 'num_object_clones': 0, 'num_object_copies': 11072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8948, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 834}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'8912", 'reported_seq': 9141, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.221531+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:06.221531+0000', 'last_peered': '2024-04-26T02:32:06.221531+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:06.221531+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8912, 'log_dups_size': 0, 'ondisk_log_size': 8912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21993734, 'num_objects': 2685, 'num_object_clones': 0, 'num_object_copies': 10740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8911, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 787}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9064", 'reported_seq': 9281, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:05.334626+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:05.334626+0000', 'last_peered': '2024-04-26T02:32:05.334626+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:05.334626+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9064, 'log_dups_size': 0, 'ondisk_log_size': 9064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22239494, 'num_objects': 2715, 'num_object_clones': 0, 'num_object_copies': 10860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9063, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 670}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9324", 'reported_seq': 9571, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:06.138841+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:06.138841+0000', 'last_peered': '2024-04-26T02:32:06.138841+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:06.138841+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9324, 'log_dups_size': 0, 'ondisk_log_size': 9324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22917644, 'num_objects': 2798, 'num_object_clones': 0, 'num_object_copies': 11192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9322, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9056", 'reported_seq': 9285, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:03.164716+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:03.164716+0000', 'last_peered': '2024-04-26T02:32:03.164716+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:03.164716+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9056, 'log_dups_size': 0, 'ondisk_log_size': 9056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22450700, 'num_objects': 2741, 'num_object_clones': 0, 'num_object_copies': 10964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9054, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:06.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:08.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:08.988 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24704/175200 objects degraded (14.100%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":525,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43800,"data_bytes":358751424,"bytes_used":2683629568,"bytes_avail":383863427072,"bytes_total":386547056640,"degraded_objects":24704,"degraded_total":175200,"degraded_ratio":0.14100456621004567,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:32:09.125 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:09.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:10.851 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:10.852 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:10.936 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-26T02:32:10.662552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353713344,"num_objects":43185,"num_object_clones":0,"num_object_copies":172740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43185,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144372,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144399,"ondisk_log_size":144399,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625084,"kb_used_data":691420,"kb_used_omap":14,"kb_used_meta":1933617,"kb_avail":374862276,"statfs":{"total":386547056640,"available":383858970624,"internally_reserved":0,"allocated":708014080,"data_stored":707199432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1980024371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,4,8,12,17,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20226048,"num_objects":-2469,"num_object_clones":0,"num_object_copies":-9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011172"},"pg_stats":[{"pgid":"1.f","version":"38'8802","reported_seq":9034,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.290222+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:06.290222+0000","last_peered":"2024-04-26T02:32:06.290222+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:06.290222+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22298624,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9148","reported_seq":9376,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:08.513735+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:08.513735+0000","last_peered":"2024-04-26T02:32:08.513735+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:08.513735+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9148,"log_dups_size":0,"ondisk_log_size":9148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21983756,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9146,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8918","reported_seq":9086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.318789+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:10.318789+0000","last_peered":"2024-04-26T02:32:10.318789+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:10.318789+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22214942,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8917,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":729}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8785","reported_seq":12810,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:09.829455+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:09.829455+0000","last_peered":"2024-04-26T02:32:09.829455+0000","last_clean":"2024-04-26T02:32:09.829455+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:09.829455+0000","last_undegraded":"2024-04-26T02:32:09.829455+0000","last_fullsized":"2024-04-26T02:32:09.829455+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:11.470557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22073868,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8783,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8872","reported_seq":9102,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.184620+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:06.184620+0000","last_peered":"2024-04-26T02:32:06.184620+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:06.184620+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8872,"log_dups_size":0,"ondisk_log_size":8872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22319628,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8870,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'8964","reported_seq":9220,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.283770+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:06.283770+0000","last_peered":"2024-04-26T02:32:06.283770+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:06.283770+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21594112,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9076","reported_seq":13327,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:08.446516+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:08.446516+0000","last_peered":"2024-04-26T02:32:08.446516+0000","last_clean":"2024-04-26T02:32:08.446516+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:08.446516+0000","last_undegraded":"2024-04-26T02:32:08.446516+0000","last_fullsized":"2024-04-26T02:32:08.446516+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:13.018682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21967372,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9305","reported_seq":9559,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:08.367389+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:08.367389+0000","last_peered":"2024-04-26T02:32:08.367389+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:08.367389+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21983756,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'8977","reported_seq":11046,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.301542+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:06.301542+0000","last_peered":"2024-04-26T02:32:06.301542+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:06.301542+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22627352,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8969","reported_seq":9143,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.379611+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:10.379611+0000","last_peered":"2024-04-26T02:32:10.379611+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:10.379611+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21254668,"num_objects":2595,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'8951","reported_seq":9183,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.396943+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:06.396943+0000","last_peered":"2024-04-26T02:32:06.396943+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:06.396943+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8951,"log_dups_size":0,"ondisk_log_size":8951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22670098,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8912","reported_seq":9141,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.221531+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:06.221531+0000","last_peered":"2024-04-26T02:32:06.221531+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:06.221531+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21993734,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9145","reported_seq":9362,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.496786+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:10.496786+0000","last_peered":"2024-04-26T02:32:10.496786+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:10.496786+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21575942,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9324","reported_seq":9571,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:06.138841+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:06.138841+0000","last_peered":"2024-04-26T02:32:06.138841+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:06.138841+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9324,"log_dups_size":0,"ondisk_log_size":9324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22917644,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9322,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9125","reported_seq":13334,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:09.898642+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:09.898642+0000","last_peered":"2024-04-26T02:32:09.898642+0000","last_clean":"2024-04-26T02:32:09.898642+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:09.898642+0000","last_undegraded":"2024-04-26T02:32:09.898642+0000","last_fullsized":"2024-04-26T02:32:09.898642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9125,"log_dups_size":0,"ondisk_log_size":9125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:13.404898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22360588,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9123,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9126","reported_seq":9355,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:08.490083+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:08.490083+0000","last_peered":"2024-04-26T02:32:08.490083+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:08.490083+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9126,"log_dups_size":0,"ondisk_log_size":9126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21877260,"num_objects":2671,"num_object_clones":0,"num_object_copies":10684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9124,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":353713344,"num_objects":43185,"num_object_clones":0,"num_object_copies":172740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43185,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144372,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":705261568,"data_stored":705261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144399,"ondisk_log_size":144399,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684168,"kb_used_data":174912,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93687672,"statfs":{"total":96636764160,"available":95936176128,"internally_reserved":0,"allocated":179109888,"data_stored":178906226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675728,"kb_used_data":171144,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93696112,"statfs":{"total":96636764160,"available":95944818688,"internally_reserved":0,"allocated":175251456,"data_stored":175047794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587028,"kb_used_data":172236,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93784812,"statfs":{"total":96636764160,"available":96035647488,"internally_reserved":0,"allocated":176369664,"data_stored":176166002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678160,"kb_used_data":173128,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93693680,"statfs":{"total":96636764160,"available":95942328320,"internally_reserved":0,"allocated":177283072,"data_stored":177079410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176529408,"data_stored":176529408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175562752,"data_stored":175562752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174854144,"data_stored":174854144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178315264,"data_stored":178315264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:13.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:15.526 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:15.576 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-26T02:32:14.663522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347954368,"num_objects":42482,"num_object_clones":0,"num_object_copies":169928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145075,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8107,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145102,"ondisk_log_size":145102,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2608464,"kb_used_data":683120,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374878896,"statfs":{"total":386547056640,"available":383875989504,"internally_reserved":0,"allocated":699514880,"data_stored":698700232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1971504691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,9,14,13,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19652608,"num_objects":-2399,"num_object_clones":0,"num_object_copies":-9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004694"},"pg_stats":[{"pgid":"1.f","version":"38'8857","reported_seq":9089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.215840+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:11.215840+0000","last_peered":"2024-04-26T02:32:11.215840+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:11.215840+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21848064,"num_objects":2667,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9216","reported_seq":9443,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.633708+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:13.633708+0000","last_peered":"2024-04-26T02:32:13.633708+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:13.633708+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21426700,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8918","reported_seq":9086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.318789+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:10.318789+0000","last_peered":"2024-04-26T02:32:10.318789+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:10.318789+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22214942,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8917,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":729}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8785","reported_seq":12810,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:09.829455+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:09.829455+0000","last_peered":"2024-04-26T02:32:09.829455+0000","last_clean":"2024-04-26T02:32:09.829455+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:09.829455+0000","last_undegraded":"2024-04-26T02:32:09.829455+0000","last_fullsized":"2024-04-26T02:32:09.829455+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:11.470557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22073868,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8783,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8925","reported_seq":9155,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.910677+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:10.910677+0000","last_peered":"2024-04-26T02:32:10.910677+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:10.910677+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21885452,"num_objects":2672,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8923,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9039","reported_seq":9295,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.360630+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:11.360630+0000","last_peered":"2024-04-26T02:32:11.360630+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:11.360630+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20979712,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9147","reported_seq":13399,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:13.536071+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:13.536071+0000","last_peered":"2024-04-26T02:32:13.536071+0000","last_clean":"2024-04-26T02:32:13.536071+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:13.536071+0000","last_undegraded":"2024-04-26T02:32:13.536071+0000","last_fullsized":"2024-04-26T02:32:13.536071+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9147,"log_dups_size":0,"ondisk_log_size":9147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:18.062574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21385740,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9368","reported_seq":9622,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.337114+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:13.337114+0000","last_peered":"2024-04-26T02:32:13.337114+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:13.337114+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21467660,"num_objects":2621,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9366,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9040","reported_seq":11194,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.362310+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:11.362310+0000","last_peered":"2024-04-26T02:32:11.362310+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:11.362310+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22111256,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'8969","reported_seq":9143,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.379611+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:10.379611+0000","last_peered":"2024-04-26T02:32:10.379611+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:10.379611+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21254668,"num_objects":2595,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":600}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9016","reported_seq":9250,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.132783+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:11.132783+0000","last_peered":"2024-04-26T02:32:11.132783+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:11.132783+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22137618,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'8974","reported_seq":9203,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.335367+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:11.335367+0000","last_peered":"2024-04-26T02:32:11.335367+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:11.335367+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21485830,"num_objects":2623,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9145","reported_seq":9362,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:10.496786+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:10.496786+0000","last_peered":"2024-04-26T02:32:10.496786+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:10.496786+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21575942,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9386","reported_seq":9633,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:11.369302+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:11.369302+0000","last_peered":"2024-04-26T02:32:11.369302+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:11.369302+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22409740,"num_objects":2736,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9384,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":757}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9125","reported_seq":13334,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:09.898642+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:09.898642+0000","last_peered":"2024-04-26T02:32:09.898642+0000","last_clean":"2024-04-26T02:32:09.898642+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:09.898642+0000","last_undegraded":"2024-04-26T02:32:09.898642+0000","last_fullsized":"2024-04-26T02:32:09.898642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9125,"log_dups_size":0,"ondisk_log_size":9125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:13.404898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22360588,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9123,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9192","reported_seq":9420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.474677+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:13.474677+0000","last_peered":"2024-04-26T02:32:13.474677+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:13.474677+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21336588,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":347954368,"num_objects":42482,"num_object_clones":0,"num_object_copies":169928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145075,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8107,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":696688640,"data_stored":696688640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145102,"ondisk_log_size":145102,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667580,"kb_used_data":170804,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93704260,"statfs":{"total":96636764160,"available":95953162240,"internally_reserved":0,"allocated":174903296,"data_stored":174699634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675728,"kb_used_data":171144,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93696112,"statfs":{"total":96636764160,"available":95944818688,"internally_reserved":0,"allocated":175251456,"data_stored":175047794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587028,"kb_used_data":172236,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93784812,"statfs":{"total":96636764160,"available":96035647488,"internally_reserved":0,"allocated":176369664,"data_stored":176166002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678128,"kb_used_data":168936,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93693712,"statfs":{"total":96636764160,"available":95942361088,"internally_reserved":0,"allocated":172990464,"data_stored":172786802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172199936,"data_stored":172199936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175562752,"data_stored":175562752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174854144,"data_stored":174854144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174071808,"data_stored":174071808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:15.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:16.926 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:16.927 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:17.052 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-26T02:32:16.664035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":341490880,"num_objects":41693,"num_object_clones":0,"num_object_copies":166772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41693,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145864,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8167,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145891,"ondisk_log_size":145891,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2630600,"kb_used_data":670568,"kb_used_omap":14,"kb_used_meta":1959985,"kb_avail":374856760,"statfs":{"total":386547056640,"available":383853322240,"internally_reserved":0,"allocated":686661632,"data_stored":685846984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2007025203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,4,16,4,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23764992,"num_objects":-2901,"num_object_clones":0,"num_object_copies":-11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004710"},"pg_stats":[{"pgid":"1.f","version":"38'8924","reported_seq":9156,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.562905+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:16.562905+0000","last_peered":"2024-04-26T02:32:16.562905+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:16.562905+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9216","reported_seq":9443,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.633708+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:13.633708+0000","last_peered":"2024-04-26T02:32:13.633708+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:13.633708+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21426700,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'8988","reported_seq":9157,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:15.495637+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:15.495637+0000","last_peered":"2024-04-26T02:32:15.495637+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:15.495637+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21641502,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8987,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8847","reported_seq":12872,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:14.890412+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:14.890412+0000","last_peered":"2024-04-26T02:32:14.890412+0000","last_clean":"2024-04-26T02:32:14.890412+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:14.890412+0000","last_undegraded":"2024-04-26T02:32:14.890412+0000","last_fullsized":"2024-04-26T02:32:14.890412+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:17.417150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21565964,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8988","reported_seq":9218,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.336774+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:16.336774+0000","last_peered":"2024-04-26T02:32:16.336774+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:16.336774+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21369356,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8986,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9107","reported_seq":9363,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.515153+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:16.515153+0000","last_peered":"2024-04-26T02:32:16.515153+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:16.515153+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9107,"log_dups_size":0,"ondisk_log_size":9107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20422656,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9147","reported_seq":13399,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:13.536071+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:13.536071+0000","last_peered":"2024-04-26T02:32:13.536071+0000","last_clean":"2024-04-26T02:32:13.536071+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:13.536071+0000","last_undegraded":"2024-04-26T02:32:13.536071+0000","last_fullsized":"2024-04-26T02:32:13.536071+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9147,"log_dups_size":0,"ondisk_log_size":9147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:18.062574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21385740,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9368","reported_seq":9622,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.337114+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:13.337114+0000","last_peered":"2024-04-26T02:32:13.337114+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:13.337114+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21467660,"num_objects":2621,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9366,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9106","reported_seq":11380,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.493745+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:16.493745+0000","last_peered":"2024-04-26T02:32:16.493745+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:16.493745+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21570584,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9103,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9033","reported_seq":9207,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:15.443780+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:15.443780+0000","last_peered":"2024-04-26T02:32:15.443780+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:15.443780+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20730380,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9031,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9067","reported_seq":9301,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.009829+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:16.009829+0000","last_peered":"2024-04-26T02:32:16.009829+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:16.009829+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21719826,"num_objects":2652,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9064,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9036","reported_seq":9265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.564452+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:16.564452+0000","last_peered":"2024-04-26T02:32:16.564452+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:16.564452+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20977926,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9213","reported_seq":9430,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:15.440701+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:15.440701+0000","last_peered":"2024-04-26T02:32:15.440701+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:15.440701+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21018886,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":521}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9460","reported_seq":9707,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.325286+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:16.325286+0000","last_peered":"2024-04-26T02:32:16.325286+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:16.325286+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9460,"log_dups_size":0,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21803532,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9199","reported_seq":13409,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:14.950349+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:14.950349+0000","last_peered":"2024-04-26T02:32:14.950349+0000","last_clean":"2024-04-26T02:32:14.950349+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:14.950349+0000","last_undegraded":"2024-04-26T02:32:14.950349+0000","last_fullsized":"2024-04-26T02:32:14.950349+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9199,"log_dups_size":0,"ondisk_log_size":9199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:18.432441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21754380,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9192","reported_seq":9420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:13.474677+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:13.474677+0000","last_peered":"2024-04-26T02:32:13.474677+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:13.474677+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21336588,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":341490880,"num_objects":41693,"num_object_clones":0,"num_object_copies":166772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41693,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145864,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8167,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":683569152,"data_stored":683569152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145891,"ondisk_log_size":145891,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671708,"kb_used_data":166612,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93700132,"statfs":{"total":96636764160,"available":95948935168,"internally_reserved":0,"allocated":170610688,"data_stored":170407026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,9,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":166988,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":170995712,"data_stored":170792050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605032,"kb_used_data":168032,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93766808,"statfs":{"total":96636764160,"available":96017211392,"internally_reserved":0,"allocated":172064768,"data_stored":171861106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678128,"kb_used_data":168936,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93693712,"statfs":{"total":96636764160,"available":95942361088,"internally_reserved":0,"allocated":172990464,"data_stored":172786802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172199936,"data_stored":172199936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171147264,"data_stored":171147264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170549248,"data_stored":170549248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169672704,"data_stored":169672704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8924", 'reported_seq': 9156, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.562905+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:16.562905+0000', 'last_peered': '2024-04-26T02:32:16.562905+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:16.562905+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8924, 'log_dups_size': 0, 'ondisk_log_size': 8924, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21299200, 'num_objects': 2600, 'num_object_clones': 0, 'num_object_copies': 10400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8924, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 673}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9216", 'reported_seq': 9443, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:13.633708+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:13.633708+0000', 'last_peered': '2024-04-26T02:32:13.633708+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:13.633708+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9216, 'log_dups_size': 0, 'ondisk_log_size': 9216, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21426700, 'num_objects': 2616, 'num_object_clones': 0, 'num_object_copies': 10464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9214, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'8988", 'reported_seq': 9157, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:15.495637+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:15.495637+0000', 'last_peered': '2024-04-26T02:32:15.495637+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:15.495637+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8988, 'log_dups_size': 0, 'ondisk_log_size': 8988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21641502, 'num_objects': 2643, 'num_object_clones': 0, 'num_object_copies': 10572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2643, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8987, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 659}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:17.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'8988", 'reported_seq': 9218, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.336774+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:16.336774+0000', 'last_peered': '2024-04-26T02:32:16.336774+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:16.336774+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8988, 'log_dups_size': 0, 'ondisk_log_size': 8988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21369356, 'num_objects': 2609, 'num_object_clones': 0, 'num_object_copies': 10436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8986, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9107", 'reported_seq': 9363, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.515153+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:16.515153+0000', 'last_peered': '2024-04-26T02:32:16.515153+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:16.515153+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9107, 'log_dups_size': 0, 'ondisk_log_size': 9107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20422656, 'num_objects': 2493, 'num_object_clones': 0, 'num_object_copies': 9972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9107, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9368", 'reported_seq': 9622, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:13.337114+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:13.337114+0000', 'last_peered': '2024-04-26T02:32:13.337114+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:13.337114+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9368, 'log_dups_size': 0, 'ondisk_log_size': 9368, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21467660, 'num_objects': 2621, 'num_object_clones': 0, 'num_object_copies': 10484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9366, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9106", 'reported_seq': 11380, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.493745+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:16.493745+0000', 'last_peered': '2024-04-26T02:32:16.493745+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:16.493745+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9106, 'log_dups_size': 0, 'ondisk_log_size': 9106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21570584, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9103, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1106, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1730}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9033", 'reported_seq': 9207, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:15.443780+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:15.443780+0000', 'last_peered': '2024-04-26T02:32:15.443780+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:15.443780+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9033, 'log_dups_size': 0, 'ondisk_log_size': 9033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20730380, 'num_objects': 2531, 'num_object_clones': 0, 'num_object_copies': 10124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9031, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:17.055 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9067", 'reported_seq': 9301, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.009829+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:16.009829+0000', 'last_peered': '2024-04-26T02:32:16.009829+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:16.009829+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9067, 'log_dups_size': 0, 'ondisk_log_size': 9067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21719826, 'num_objects': 2652, 'num_object_clones': 0, 'num_object_copies': 10608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9064, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 718}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9036", 'reported_seq': 9265, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.564452+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:16.564452+0000', 'last_peered': '2024-04-26T02:32:16.564452+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:16.564452+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9036, 'log_dups_size': 0, 'ondisk_log_size': 9036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20977926, 'num_objects': 2561, 'num_object_clones': 0, 'num_object_copies': 10244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9035, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9213", 'reported_seq': 9430, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:15.440701+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:15.440701+0000', 'last_peered': '2024-04-26T02:32:15.440701+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:15.440701+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9213, 'log_dups_size': 0, 'ondisk_log_size': 9213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21018886, 'num_objects': 2566, 'num_object_clones': 0, 'num_object_copies': 10264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9212, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9460", 'reported_seq': 9707, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:16.325286+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:16.325286+0000', 'last_peered': '2024-04-26T02:32:16.325286+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:16.325286+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9460, 'log_dups_size': 0, 'ondisk_log_size': 9460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21803532, 'num_objects': 2662, 'num_object_clones': 0, 'num_object_copies': 10648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9458, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:17.056 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9192", 'reported_seq': 9420, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:13.474677+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:13.474677+0000', 'last_peered': '2024-04-26T02:32:13.474677+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:13.474677+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9192, 'log_dups_size': 0, 'ondisk_log_size': 9192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21336588, 'num_objects': 2605, 'num_object_clones': 0, 'num_object_copies': 10420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9190, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 618}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:17.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:19.119 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:19.119 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24602/166772 objects degraded (14.752%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":535,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41693,"data_bytes":341490880,"bytes_used":2693734400,"bytes_avail":383853322240,"bytes_total":386547056640,"degraded_objects":24602,"degraded_total":166772,"degraded_ratio":0.14751876813853645,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:32:19.167 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:19.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:21.015 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:21.016 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:21.119 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-26T02:32:20.667028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":336641216,"num_objects":41101,"num_object_clones":0,"num_object_copies":164404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146456,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8167,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146483,"ondisk_log_size":146483,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2585032,"kb_used_data":658272,"kb_used_omap":14,"kb_used_meta":1926705,"kb_avail":374902328,"statfs":{"total":386547056640,"available":383899983872,"internally_reserved":0,"allocated":674070528,"data_stored":673255880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1972946483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,3,17,7,11,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19791872,"num_objects":-2416,"num_object_clones":0,"num_object_copies":-9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006662"},"pg_stats":[{"pgid":"1.f","version":"38'8924","reported_seq":9156,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.562905+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:16.562905+0000","last_peered":"2024-04-26T02:32:16.562905+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:16.562905+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9277","reported_seq":9505,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:18.437300+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:18.437300+0000","last_peered":"2024-04-26T02:32:18.437300+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:18.437300+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9277,"log_dups_size":0,"ondisk_log_size":9277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20926988,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":555}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9049","reported_seq":9218,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.326751+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:20.326751+0000","last_peered":"2024-04-26T02:32:20.326751+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:20.326751+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9049,"log_dups_size":0,"ondisk_log_size":9049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21141790,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9048,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":598}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8917","reported_seq":12942,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:20.040167+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:20.040167+0000","last_peered":"2024-04-26T02:32:20.040167+0000","last_clean":"2024-04-26T02:32:20.040167+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:20.040167+0000","last_undegraded":"2024-04-26T02:32:20.040167+0000","last_fullsized":"2024-04-26T02:32:20.040167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:23.400740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20992524,"num_objects":2563,"num_object_clones":0,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'8988","reported_seq":9218,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.336774+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:16.336774+0000","last_peered":"2024-04-26T02:32:16.336774+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:16.336774+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21369356,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8986,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9107","reported_seq":9363,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.515153+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:16.515153+0000","last_peered":"2024-04-26T02:32:16.515153+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:16.515153+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9107,"log_dups_size":0,"ondisk_log_size":9107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20422656,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9208","reported_seq":13460,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:18.553959+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:18.553959+0000","last_peered":"2024-04-26T02:32:18.553959+0000","last_clean":"2024-04-26T02:32:18.553959+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:18.553959+0000","last_undegraded":"2024-04-26T02:32:18.553959+0000","last_fullsized":"2024-04-26T02:32:18.553959+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9208,"log_dups_size":0,"ondisk_log_size":9208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:23.070575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20886028,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9206,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9441","reported_seq":9695,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:18.461529+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:18.461529+0000","last_peered":"2024-04-26T02:32:18.461529+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:18.461529+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9441,"log_dups_size":0,"ondisk_log_size":9441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20869644,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9106","reported_seq":11380,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.493745+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:16.493745+0000","last_peered":"2024-04-26T02:32:16.493745+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:16.493745+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21570584,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9103,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9091","reported_seq":9265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.469589+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:20.469589+0000","last_peered":"2024-04-26T02:32:20.469589+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:20.469589+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20255244,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9067","reported_seq":9301,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.009829+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:16.009829+0000","last_peered":"2024-04-26T02:32:16.009829+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:16.009829+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21719826,"num_objects":2652,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9064,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9036","reported_seq":9265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.564452+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:16.564452+0000","last_peered":"2024-04-26T02:32:16.564452+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:16.564452+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20977926,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9276","reported_seq":9493,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.338679+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:20.338679+0000","last_peered":"2024-04-26T02:32:20.338679+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:20.338679+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20502790,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9460","reported_seq":9707,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:16.325286+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:16.325286+0000","last_peered":"2024-04-26T02:32:16.325286+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:16.325286+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9460,"log_dups_size":0,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21803532,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9269","reported_seq":13479,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:20.027163+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:20.027163+0000","last_peered":"2024-04-26T02:32:20.027163+0000","last_clean":"2024-04-26T02:32:20.027163+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:20.027163+0000","last_undegraded":"2024-04-26T02:32:20.027163+0000","last_fullsized":"2024-04-26T02:32:20.027163+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:24.390943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21180940,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9267","reported_seq":9496,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:18.655835+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:18.655835+0000","last_peered":"2024-04-26T02:32:18.655835+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:18.655835+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9267,"log_dups_size":0,"ondisk_log_size":9267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722188,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9265,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":336641216,"num_objects":41101,"num_object_clones":0,"num_object_copies":164404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146456,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8167,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":670814208,"data_stored":670814208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146483,"ondisk_log_size":146483,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671708,"kb_used_data":166612,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93700132,"statfs":{"total":96636764160,"available":95948935168,"internally_reserved":0,"allocated":170610688,"data_stored":170407026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,9,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659124,"kb_used_data":162852,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93712716,"statfs":{"total":96636764160,"available":95961821184,"internally_reserved":0,"allocated":166760448,"data_stored":166556786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592640,"kb_used_data":163960,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93779200,"statfs":{"total":96636764160,"available":96029900800,"internally_reserved":0,"allocated":167895040,"data_stored":167691378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,4,4,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661560,"kb_used_data":164848,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93710280,"statfs":{"total":96636764160,"available":95959326720,"internally_reserved":0,"allocated":168804352,"data_stored":168600690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167907328,"data_stored":167907328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166895616,"data_stored":166895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166338560,"data_stored":166338560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169672704,"data_stored":169672704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:24.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:25.645 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:25.645 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:25.733 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-04-26T02:32:24.668079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330857664,"num_objects":40395,"num_object_clones":0,"num_object_copies":161580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147162,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8208,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147189,"ondisk_log_size":147189,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2584968,"kb_used_data":649888,"kb_used_omap":14,"kb_used_meta":1935025,"kb_avail":374902392,"statfs":{"total":386547056640,"available":383900049408,"internally_reserved":0,"allocated":665485312,"data_stored":664670664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":1981466163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,14,17,19,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19292160,"num_objects":-2355,"num_object_clones":0,"num_object_copies":-9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005049"},"pg_stats":[{"pgid":"1.f","version":"38'8994","reported_seq":9226,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.558860+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:21.558860+0000","last_peered":"2024-04-26T02:32:21.558860+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:21.558860+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20725760,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8994,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9334","reported_seq":9562,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.573511+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:23.573511+0000","last_peered":"2024-04-26T02:32:23.573511+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:23.573511+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460044,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9332,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9049","reported_seq":9218,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.326751+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:20.326751+0000","last_peered":"2024-04-26T02:32:20.326751+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:20.326751+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9049,"log_dups_size":0,"ondisk_log_size":9049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21141790,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9048,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":598}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8917","reported_seq":12942,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:20.040167+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:20.040167+0000","last_peered":"2024-04-26T02:32:20.040167+0000","last_clean":"2024-04-26T02:32:20.040167+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:20.040167+0000","last_undegraded":"2024-04-26T02:32:20.040167+0000","last_fullsized":"2024-04-26T02:32:20.040167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:23.400740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20992524,"num_objects":2563,"num_object_clones":0,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8915,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9043","reported_seq":9274,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.669142+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:21.669142+0000","last_peered":"2024-04-26T02:32:21.669142+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:21.669142+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918796,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9167","reported_seq":9423,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.633754+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:21.633754+0000","last_peered":"2024-04-26T02:32:21.633754+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:21.633754+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19931136,"num_objects":2433,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9284","reported_seq":13536,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:23.727350+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:23.727350+0000","last_peered":"2024-04-26T02:32:23.727350+0000","last_clean":"2024-04-26T02:32:23.727350+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:23.727350+0000","last_undegraded":"2024-04-26T02:32:23.727350+0000","last_fullsized":"2024-04-26T02:32:23.727350+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:28.175384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20263436,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9282,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9488","reported_seq":9742,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.418753+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:23.418753+0000","last_peered":"2024-04-26T02:32:23.418753+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:23.418753+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20484620,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9486,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9181","reported_seq":11537,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.652869+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:21.652869+0000","last_peered":"2024-04-26T02:32:21.652869+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:21.652869+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20956184,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9178,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9091","reported_seq":9265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.469589+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:20.469589+0000","last_peered":"2024-04-26T02:32:20.469589+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:20.469589+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20255244,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9147","reported_seq":9381,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.583781+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:21.583781+0000","last_peered":"2024-04-26T02:32:21.583781+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:21.583781+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9147,"log_dups_size":0,"ondisk_log_size":9147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21064466,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9093","reported_seq":9321,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.604624+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:21.604624+0000","last_peered":"2024-04-26T02:32:21.604624+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:21.604624+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20510982,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9276","reported_seq":9493,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:20.338679+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:20.338679+0000","last_peered":"2024-04-26T02:32:20.338679+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:20.338679+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20502790,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9528","reported_seq":9774,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.598828+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:21.598828+0000","last_peered":"2024-04-26T02:32:21.598828+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:21.598828+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21246476,"num_objects":2594,"num_object_clones":0,"num_object_copies":10376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9269","reported_seq":13479,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:20.027163+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:20.027163+0000","last_peered":"2024-04-26T02:32:20.027163+0000","last_clean":"2024-04-26T02:32:20.027163+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:20.027163+0000","last_undegraded":"2024-04-26T02:32:20.027163+0000","last_fullsized":"2024-04-26T02:32:20.027163+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:24.390943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21180940,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9328","reported_seq":9557,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.756297+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:23.756297+0000","last_peered":"2024-04-26T02:32:23.756297+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:23.756297+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20222476,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9326,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":330857664,"num_objects":40395,"num_object_clones":0,"num_object_copies":161580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147162,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8208,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662147072,"data_stored":662147072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147189,"ondisk_log_size":147189,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671760,"kb_used_data":162504,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93700080,"statfs":{"total":96636764160,"available":95948881920,"internally_reserved":0,"allocated":166404096,"data_stored":166200434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,9,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659124,"kb_used_data":162852,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93712716,"statfs":{"total":96636764160,"available":95961821184,"internally_reserved":0,"allocated":166760448,"data_stored":166556786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592640,"kb_used_data":163960,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93779200,"statfs":{"total":96636764160,"available":96029900800,"internally_reserved":0,"allocated":167895040,"data_stored":167691378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,4,4,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661444,"kb_used_data":160572,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93710396,"statfs":{"total":96636764160,"available":95959445504,"internally_reserved":0,"allocated":164425728,"data_stored":164222066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163524608,"data_stored":163524608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":166895616,"data_stored":166895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166338560,"data_stored":166338560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165388288,"data_stored":165388288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:25.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:27.374 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:27.375 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:27.467 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-04-26T02:32:26.668667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328162496,"num_objects":40066,"num_object_clones":0,"num_object_copies":160264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147491,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8208,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147518,"ondisk_log_size":147518,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607188,"kb_used_data":641588,"kb_used_omap":14,"kb_used_meta":1965553,"kb_avail":374880172,"statfs":{"total":386547056640,"available":383877296128,"internally_reserved":0,"allocated":656986112,"data_stored":656171464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2012726835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,9,20,16,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19791872,"num_objects":-2416,"num_object_clones":0,"num_object_copies":-9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005145"},"pg_stats":[{"pgid":"1.f","version":"38'8994","reported_seq":9226,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.558860+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:21.558860+0000","last_peered":"2024-04-26T02:32:21.558860+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:21.558860+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20725760,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8994,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9334","reported_seq":9562,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.573511+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:23.573511+0000","last_peered":"2024-04-26T02:32:23.573511+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:23.573511+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460044,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9332,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9130","reported_seq":9297,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:25.547885+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:25.547885+0000","last_peered":"2024-04-26T02:32:25.547885+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:25.547885+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9130,"log_dups_size":0,"ondisk_log_size":9130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20478238,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9129,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'8977","reported_seq":13002,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:24.943831+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:24.943831+0000","last_peered":"2024-04-26T02:32:24.943831+0000","last_clean":"2024-04-26T02:32:24.943831+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:24.943831+0000","last_undegraded":"2024-04-26T02:32:24.943831+0000","last_fullsized":"2024-04-26T02:32:24.943831+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.283771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20501004,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9043","reported_seq":9274,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.669142+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:21.669142+0000","last_peered":"2024-04-26T02:32:21.669142+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:21.669142+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918796,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9167","reported_seq":9423,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.633754+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:21.633754+0000","last_peered":"2024-04-26T02:32:21.633754+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:21.633754+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19931136,"num_objects":2433,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9284","reported_seq":13536,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:23.727350+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:23.727350+0000","last_peered":"2024-04-26T02:32:23.727350+0000","last_clean":"2024-04-26T02:32:23.727350+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:23.727350+0000","last_undegraded":"2024-04-26T02:32:23.727350+0000","last_fullsized":"2024-04-26T02:32:23.727350+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:28.175384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20263436,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9282,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9488","reported_seq":9742,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.418753+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:23.418753+0000","last_peered":"2024-04-26T02:32:23.418753+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:23.418753+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20484620,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9486,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9181","reported_seq":11537,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.652869+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:21.652869+0000","last_peered":"2024-04-26T02:32:21.652869+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:21.652869+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20956184,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9178,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9152","reported_seq":9326,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:25.586656+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:25.586656+0000","last_peered":"2024-04-26T02:32:25.586656+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:25.586656+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9152,"log_dups_size":0,"ondisk_log_size":9152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19755532,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9147","reported_seq":9381,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.583781+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:21.583781+0000","last_peered":"2024-04-26T02:32:21.583781+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:21.583781+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9147,"log_dups_size":0,"ondisk_log_size":9147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21064466,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":638}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9093","reported_seq":9321,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.604624+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:21.604624+0000","last_peered":"2024-04-26T02:32:21.604624+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:21.604624+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20510982,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9332","reported_seq":9549,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:25.242906+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:25.242906+0000","last_peered":"2024-04-26T02:32:25.242906+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:25.242906+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9332,"log_dups_size":0,"ondisk_log_size":9332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20044038,"num_objects":2447,"num_object_clones":0,"num_object_copies":9788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9528","reported_seq":9774,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:21.598828+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:21.598828+0000","last_peered":"2024-04-26T02:32:21.598828+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:21.598828+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21246476,"num_objects":2594,"num_object_clones":0,"num_object_copies":10376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9340","reported_seq":13550,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:25.058434+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:25.058434+0000","last_peered":"2024-04-26T02:32:25.058434+0000","last_clean":"2024-04-26T02:32:25.058434+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:25.058434+0000","last_undegraded":"2024-04-26T02:32:25.058434+0000","last_fullsized":"2024-04-26T02:32:25.058434+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20599308,"num_objects":2515,"num_object_clones":0,"num_object_copies":10060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9328","reported_seq":9557,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:23.756297+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:23.756297+0000","last_peered":"2024-04-26T02:32:23.756297+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:23.756297+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20222476,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9326,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328162496,"num_objects":40066,"num_object_clones":0,"num_object_copies":160264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147491,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8208,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":653524992,"data_stored":653524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147518,"ondisk_log_size":147518,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671760,"kb_used_data":162504,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93700080,"statfs":{"total":96636764160,"available":95948881920,"internally_reserved":0,"allocated":166404096,"data_stored":166200434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,9,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681292,"kb_used_data":158660,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93690548,"statfs":{"total":96636764160,"available":95939121152,"internally_reserved":0,"allocated":162467840,"data_stored":162264178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592692,"kb_used_data":159852,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93779148,"statfs":{"total":96636764160,"available":96029847552,"internally_reserved":0,"allocated":163688448,"data_stored":163484786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661444,"kb_used_data":160572,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93710396,"statfs":{"total":96636764160,"available":95959445504,"internally_reserved":0,"allocated":164425728,"data_stored":164222066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163524608,"data_stored":163524608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162545664,"data_stored":162545664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162066432,"data_stored":162066432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165388288,"data_stored":165388288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:27.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:27.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'8994", 'reported_seq': 9226, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.558860+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:21.558860+0000', 'last_peered': '2024-04-26T02:32:21.558860+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:21.558860+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 8994, 'log_dups_size': 0, 'ondisk_log_size': 8994, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20725760, 'num_objects': 2530, 'num_object_clones': 0, 'num_object_copies': 10120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8994, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:27.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9334", 'reported_seq': 9562, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:23.573511+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:23.573511+0000', 'last_peered': '2024-04-26T02:32:23.573511+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:23.573511+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9334, 'log_dups_size': 0, 'ondisk_log_size': 9334, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20460044, 'num_objects': 2498, 'num_object_clones': 0, 'num_object_copies': 9992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9332, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9130", 'reported_seq': 9297, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:25.547885+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:25.547885+0000', 'last_peered': '2024-04-26T02:32:25.547885+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:25.547885+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9130, 'log_dups_size': 0, 'ondisk_log_size': 9130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20478238, 'num_objects': 2501, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2501, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9129, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 517}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9043", 'reported_seq': 9274, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.669142+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:21.669142+0000', 'last_peered': '2024-04-26T02:32:21.669142+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:21.669142+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9043, 'log_dups_size': 0, 'ondisk_log_size': 9043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20918796, 'num_objects': 2554, 'num_object_clones': 0, 'num_object_copies': 10216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9041, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 625}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9167", 'reported_seq': 9423, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.633754+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:21.633754+0000', 'last_peered': '2024-04-26T02:32:21.633754+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:21.633754+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9167, 'log_dups_size': 0, 'ondisk_log_size': 9167, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19931136, 'num_objects': 2433, 'num_object_clones': 0, 'num_object_copies': 9732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9167, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 483}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9488", 'reported_seq': 9742, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:23.418753+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:23.418753+0000', 'last_peered': '2024-04-26T02:32:23.418753+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:23.418753+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9488, 'log_dups_size': 0, 'ondisk_log_size': 9488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20484620, 'num_objects': 2501, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9486, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:27.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9181", 'reported_seq': 11537, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.652869+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:21.652869+0000', 'last_peered': '2024-04-26T02:32:21.652869+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:21.652869+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9181, 'log_dups_size': 0, 'ondisk_log_size': 9181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20956184, 'num_objects': 2559, 'num_object_clones': 0, 'num_object_copies': 10236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9178, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1147, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9152", 'reported_seq': 9326, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:25.586656+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:25.586656+0000', 'last_peered': '2024-04-26T02:32:25.586656+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:25.586656+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9152, 'log_dups_size': 0, 'ondisk_log_size': 9152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19755532, 'num_objects': 2412, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9150, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 417}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9147", 'reported_seq': 9381, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.583781+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:21.583781+0000', 'last_peered': '2024-04-26T02:32:21.583781+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:21.583781+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9147, 'log_dups_size': 0, 'ondisk_log_size': 9147, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21064466, 'num_objects': 2572, 'num_object_clones': 0, 'num_object_copies': 10288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9144, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 638}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9093", 'reported_seq': 9321, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.604624+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:21.604624+0000', 'last_peered': '2024-04-26T02:32:21.604624+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:21.604624+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9093, 'log_dups_size': 0, 'ondisk_log_size': 9093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20510982, 'num_objects': 2504, 'num_object_clones': 0, 'num_object_copies': 10016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9092, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9332", 'reported_seq': 9549, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:25.242906+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:25.242906+0000', 'last_peered': '2024-04-26T02:32:25.242906+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:25.242906+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9332, 'log_dups_size': 0, 'ondisk_log_size': 9332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20044038, 'num_objects': 2447, 'num_object_clones': 0, 'num_object_copies': 9788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9331, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9528", 'reported_seq': 9774, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:21.598828+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:21.598828+0000', 'last_peered': '2024-04-26T02:32:21.598828+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:21.598828+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9528, 'log_dups_size': 0, 'ondisk_log_size': 9528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21246476, 'num_objects': 2594, 'num_object_clones': 0, 'num_object_copies': 10376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9526, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:27.470 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:27.471 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9328", 'reported_seq': 9557, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:23.756297+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:23.756297+0000', 'last_peered': '2024-04-26T02:32:23.756297+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:23.756297+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9328, 'log_dups_size': 0, 'ondisk_log_size': 9328, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20222476, 'num_objects': 2469, 'num_object_clones': 0, 'num_object_copies': 9876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9326, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:27.471 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24501/158360 objects degraded (15.472%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":546,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39590,"data_bytes":324263104,"bytes_used":2671194112,"bytes_avail":383875862528,"bytes_total":386547056640,"degraded_objects":24501,"degraded_total":158360,"degraded_ratio":0.15471710027784794,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:32:29.820 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:29.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:31.583 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:31.583 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:31.653 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-04-26T02:32:30.669671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":319544512,"num_objects":39014,"num_object_clones":0,"num_object_copies":156056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39014,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148543,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8268,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148570,"ondisk_log_size":148570,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626360,"kb_used_data":624664,"kb_used_omap":14,"kb_used_meta":2001649,"kb_avail":374861000,"statfs":{"total":386547056640,"available":383857664000,"internally_reserved":0,"allocated":639655936,"data_stored":638841288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2049689139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,15,9,15,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21946368,"num_objects":-2679,"num_object_clones":0,"num_object_copies":-10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2679,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003120"},"pg_stats":[{"pgid":"1.f","version":"38'9057","reported_seq":9289,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.818755+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:26.818755+0000","last_peered":"2024-04-26T02:32:26.818755+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:26.818755+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20209664,"num_objects":2467,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9390","reported_seq":9618,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:28.625495+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:28.625495+0000","last_peered":"2024-04-26T02:32:28.625495+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:28.625495+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20001292,"num_objects":2442,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9388,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":442}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9200","reported_seq":9369,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.558034+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:30.558034+0000","last_peered":"2024-04-26T02:32:30.558034+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:30.558034+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19904798,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9199,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":447}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9034","reported_seq":13059,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:30.218230+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:30.218230+0000","last_peered":"2024-04-26T02:32:30.218230+0000","last_clean":"2024-04-26T02:32:30.218230+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:30.218230+0000","last_undegraded":"2024-04-26T02:32:30.218230+0000","last_fullsized":"2024-04-26T02:32:30.218230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:34.331942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20034060,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9121","reported_seq":9349,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.671061+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:26.671061+0000","last_peered":"2024-04-26T02:32:26.671061+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:26.671061+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20279820,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9119,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9226","reported_seq":9482,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.209046+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:26.209046+0000","last_peered":"2024-04-26T02:32:26.209046+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:26.209046+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19447808,"num_objects":2374,"num_object_clones":0,"num_object_copies":9496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":424}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9349","reported_seq":13601,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:28.730749+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:28.730749+0000","last_peered":"2024-04-26T02:32:28.730749+0000","last_clean":"2024-04-26T02:32:28.730749+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:28.730749+0000","last_undegraded":"2024-04-26T02:32:28.730749+0000","last_fullsized":"2024-04-26T02:32:28.730749+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19730956,"num_objects":2409,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9555","reported_seq":9809,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:28.768616+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:28.768616+0000","last_peered":"2024-04-26T02:32:28.768616+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:28.768616+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9555,"log_dups_size":0,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19935756,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":382}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9243","reported_seq":11719,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.742126+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:26.742126+0000","last_peered":"2024-04-26T02:32:26.742126+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:26.742126+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20448280,"num_objects":2497,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9222","reported_seq":9395,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.423767+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:30.423767+0000","last_peered":"2024-04-26T02:32:30.423767+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:30.423767+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182092,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9229","reported_seq":9463,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.745583+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:26.745583+0000","last_peered":"2024-04-26T02:32:26.745583+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:26.745583+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9229,"log_dups_size":0,"ondisk_log_size":9229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20392722,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":556}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9160","reported_seq":9389,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.485059+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:26.485059+0000","last_peered":"2024-04-26T02:32:26.485059+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:26.485059+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19962118,"num_objects":2437,"num_object_clones":0,"num_object_copies":9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":539}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9393","reported_seq":9610,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.582550+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:30.582550+0000","last_peered":"2024-04-26T02:32:30.582550+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:30.582550+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19544326,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9392,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":341}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9593","reported_seq":9840,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:26.556529+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:26.556529+0000","last_peered":"2024-04-26T02:32:26.556529+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:26.556529+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20713996,"num_objects":2529,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9591,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9402","reported_seq":13612,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:30.172161+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:30.172161+0000","last_peered":"2024-04-26T02:32:30.172161+0000","last_clean":"2024-04-26T02:32:30.172161+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:30.172161+0000","last_undegraded":"2024-04-26T02:32:30.172161+0000","last_fullsized":"2024-04-26T02:32:30.172161+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.317344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20091404,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9396","reported_seq":9624,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:28.785945+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:28.785945+0000","last_peered":"2024-04-26T02:32:28.785945+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:28.785945+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19665420,"num_objects":2401,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":319544512,"num_objects":39014,"num_object_clones":0,"num_object_copies":156056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39014,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148543,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8268,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":636780544,"data_stored":636780544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148570,"ondisk_log_size":148570,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673160,"kb_used_data":158336,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93698680,"statfs":{"total":96636764160,"available":95947448320,"internally_reserved":0,"allocated":162136064,"data_stored":161932402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":527167184},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,7,4,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681492,"kb_used_data":154700,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93690348,"statfs":{"total":96636764160,"available":95938916352,"internally_reserved":0,"allocated":158412800,"data_stored":158209138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591988,"kb_used_data":154988,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93779852,"statfs":{"total":96636764160,"available":96030568448,"internally_reserved":0,"allocated":158707712,"data_stored":158504050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9,0,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679720,"kb_used_data":156640,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93692120,"statfs":{"total":96636764160,"available":95940730880,"internally_reserved":0,"allocated":160399360,"data_stored":160195698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159416320,"data_stored":159416320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158355456,"data_stored":158355456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157941760,"data_stored":157941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161067008,"data_stored":161067008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:34.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:35.990 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:35.991 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:36.076 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-04-26T02:32:34.670675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313556160,"num_objects":38283,"num_object_clones":0,"num_object_copies":153132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149274,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8325,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149301,"ondisk_log_size":149301,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626536,"kb_used_data":616516,"kb_used_omap":14,"kb_used_meta":2009969,"kb_avail":374860824,"statfs":{"total":386547056640,"available":383857483776,"internally_reserved":0,"allocated":631312384,"data_stored":630497736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2058208819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,16,12,18,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19275776,"num_objects":-2353,"num_object_clones":0,"num_object_copies":-9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003089"},"pg_stats":[{"pgid":"1.f","version":"38'9121","reported_seq":9353,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.794751+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:31.794751+0000","last_peered":"2024-04-26T02:32:31.794751+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:31.794751+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19685376,"num_objects":2403,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9457","reported_seq":9684,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.871662+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:33.871662+0000","last_peered":"2024-04-26T02:32:33.871662+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:33.871662+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9457,"log_dups_size":0,"ondisk_log_size":9457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19452428,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9200","reported_seq":9369,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.558034+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:30.558034+0000","last_peered":"2024-04-26T02:32:30.558034+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:30.558034+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9200,"log_dups_size":0,"ondisk_log_size":9200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19904798,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9199,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":447}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9034","reported_seq":13059,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:30.218230+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:30.218230+0000","last_peered":"2024-04-26T02:32:30.218230+0000","last_clean":"2024-04-26T02:32:30.218230+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:30.218230+0000","last_undegraded":"2024-04-26T02:32:30.218230+0000","last_fullsized":"2024-04-26T02:32:30.218230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:34.331942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20034060,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9178","reported_seq":9409,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.564621+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:31.564621+0000","last_peered":"2024-04-26T02:32:31.564621+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:31.564621+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9178,"log_dups_size":0,"ondisk_log_size":9178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19812876,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9291","reported_seq":9547,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.460476+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:31.460476+0000","last_peered":"2024-04-26T02:32:31.460476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:31.460476+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18915328,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9291,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9419","reported_seq":13672,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:33.887766+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:33.887766+0000","last_peered":"2024-04-26T02:32:33.887766+0000","last_clean":"2024-04-26T02:32:33.887766+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:33.887766+0000","last_undegraded":"2024-04-26T02:32:33.887766+0000","last_fullsized":"2024-04-26T02:32:33.887766+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:34.118390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19157516,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9629","reported_seq":9883,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.751382+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:33.751382+0000","last_peered":"2024-04-26T02:32:33.751382+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:33.751382+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19329548,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9302","reported_seq":11892,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.832342+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:31.832342+0000","last_peered":"2024-04-26T02:32:31.832342+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:31.832342+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9302,"log_dups_size":0,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19964952,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9222","reported_seq":9395,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.423767+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:30.423767+0000","last_peered":"2024-04-26T02:32:30.423767+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:30.423767+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182092,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9295","reported_seq":9529,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.853960+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:31.853960+0000","last_peered":"2024-04-26T02:32:31.853960+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:31.853960+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19852050,"num_objects":2424,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9232","reported_seq":9461,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.680235+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:31.680235+0000","last_peered":"2024-04-26T02:32:31.680235+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:31.680235+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19372294,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9231,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9393","reported_seq":9610,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:30.582550+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:30.582550+0000","last_peered":"2024-04-26T02:32:30.582550+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:30.582550+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19544326,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9392,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":341}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9670","reported_seq":9917,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.843552+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:31.843552+0000","last_peered":"2024-04-26T02:32:31.843552+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:31.843552+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20083212,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":473}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9402","reported_seq":13612,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:30.172161+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:30.172161+0000","last_peered":"2024-04-26T02:32:30.172161+0000","last_clean":"2024-04-26T02:32:30.172161+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:30.172161+0000","last_undegraded":"2024-04-26T02:32:30.172161+0000","last_fullsized":"2024-04-26T02:32:30.172161+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.317344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20091404,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9456","reported_seq":9685,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.846382+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:33.846382+0000","last_peered":"2024-04-26T02:32:33.846382+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:33.846382+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9456,"log_dups_size":0,"ondisk_log_size":9456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173900,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":313556160,"num_objects":38283,"num_object_clones":0,"num_object_copies":153132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149274,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8325,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":628322304,"data_stored":628322304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149301,"ondisk_log_size":149301,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673272,"kb_used_data":154288,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93698568,"statfs":{"total":96636764160,"available":95947333632,"internally_reserved":0,"allocated":157990912,"data_stored":157787250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":531427024},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681492,"kb_used_data":154700,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93690348,"statfs":{"total":96636764160,"available":95938916352,"internally_reserved":0,"allocated":158412800,"data_stored":158209138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591988,"kb_used_data":154988,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93779852,"statfs":{"total":96636764160,"available":96030568448,"internally_reserved":0,"allocated":158707712,"data_stored":158504050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9,0,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679784,"kb_used_data":152540,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93692056,"statfs":{"total":96636764160,"available":95940665344,"internally_reserved":0,"allocated":156200960,"data_stored":155997298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155156480,"data_stored":155156480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158355456,"data_stored":158355456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157941760,"data_stored":157941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":156868608,"data_stored":156868608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:36.077 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:37.830 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:38.010 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-26T02:32:36.671163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":310967488,"num_objects":37967,"num_object_clones":0,"num_object_copies":151868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149590,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8325,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149617,"ondisk_log_size":149617,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2597440,"kb_used_data":608208,"kb_used_omap":14,"kb_used_meta":1989169,"kb_avail":374889920,"statfs":{"total":386547056640,"available":383887278080,"internally_reserved":0,"allocated":622804992,"data_stored":621990344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2036909619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,7,25,17,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19890176,"num_objects":-2428,"num_object_clones":0,"num_object_copies":-9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2428,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.f","version":"38'9121","reported_seq":9353,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.794751+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:31.794751+0000","last_peered":"2024-04-26T02:32:31.794751+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:31.794751+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19685376,"num_objects":2403,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9457","reported_seq":9684,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.871662+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:33.871662+0000","last_peered":"2024-04-26T02:32:33.871662+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:33.871662+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9457,"log_dups_size":0,"ondisk_log_size":9457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19452428,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9271","reported_seq":9440,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.558419+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:35.558419+0000","last_peered":"2024-04-26T02:32:35.558419+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:35.558419+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19323166,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9270,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9102","reported_seq":13127,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:35.432593+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:35.432593+0000","last_peered":"2024-04-26T02:32:35.432593+0000","last_clean":"2024-04-26T02:32:35.432593+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:35.432593+0000","last_undegraded":"2024-04-26T02:32:35.432593+0000","last_fullsized":"2024-04-26T02:32:35.432593+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9102,"log_dups_size":0,"ondisk_log_size":9102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:39.503220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19477004,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9100,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9178","reported_seq":9409,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.564621+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:31.564621+0000","last_peered":"2024-04-26T02:32:31.564621+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:31.564621+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9178,"log_dups_size":0,"ondisk_log_size":9178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19812876,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9291","reported_seq":9547,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.460476+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:31.460476+0000","last_peered":"2024-04-26T02:32:31.460476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:31.460476+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18915328,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9291,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9419","reported_seq":13672,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:33.887766+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:33.887766+0000","last_peered":"2024-04-26T02:32:33.887766+0000","last_clean":"2024-04-26T02:32:33.887766+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:33.887766+0000","last_undegraded":"2024-04-26T02:32:33.887766+0000","last_fullsized":"2024-04-26T02:32:33.887766+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:34.118390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19157516,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9629","reported_seq":9883,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.751382+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:33.751382+0000","last_peered":"2024-04-26T02:32:33.751382+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:33.751382+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19329548,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9302","reported_seq":11892,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.832342+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:31.832342+0000","last_peered":"2024-04-26T02:32:31.832342+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:31.832342+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9302,"log_dups_size":0,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19964952,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9291","reported_seq":9465,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.549295+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:35.549295+0000","last_peered":"2024-04-26T02:32:35.549295+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:35.549295+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18616844,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":278}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9295","reported_seq":9529,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.853960+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:31.853960+0000","last_peered":"2024-04-26T02:32:31.853960+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:31.853960+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19852050,"num_objects":2424,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9232","reported_seq":9461,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.680235+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:31.680235+0000","last_peered":"2024-04-26T02:32:31.680235+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:31.680235+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19372294,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9231,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":467}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9440","reported_seq":9657,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.283605+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:35.283605+0000","last_peered":"2024-04-26T02:32:35.283605+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:35.283605+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19159302,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":294}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9670","reported_seq":9917,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:31.843552+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:31.843552+0000","last_peered":"2024-04-26T02:32:31.843552+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:31.843552+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20083212,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9668,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":473}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9463","reported_seq":13673,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:35.520510+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:35.520510+0000","last_peered":"2024-04-26T02:32:35.520510+0000","last_clean":"2024-04-26T02:32:35.520510+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:35.520510+0000","last_undegraded":"2024-04-26T02:32:35.520510+0000","last_fullsized":"2024-04-26T02:32:35.520510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:40.466518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19591692,"num_objects":2392,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9456","reported_seq":9685,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:33.846382+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:33.846382+0000","last_peered":"2024-04-26T02:32:33.846382+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:33.846382+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9456,"log_dups_size":0,"ondisk_log_size":9456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173900,"num_objects":2341,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":310967488,"num_objects":37967,"num_object_clones":0,"num_object_copies":151868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149590,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8325,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":619778048,"data_stored":619778048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149617,"ondisk_log_size":149617,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673272,"kb_used_data":154288,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93698568,"statfs":{"total":96636764160,"available":95947333632,"internally_reserved":0,"allocated":157990912,"data_stored":157787250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":531427024},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664900,"kb_used_data":150576,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93706940,"statfs":{"total":96636764160,"available":95955906560,"internally_reserved":0,"allocated":154189824,"data_stored":153986162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579484,"kb_used_data":150804,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93792356,"statfs":{"total":96636764160,"available":96043372544,"internally_reserved":0,"allocated":154423296,"data_stored":154219634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679784,"kb_used_data":152540,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93692056,"statfs":{"total":96636764160,"available":95940665344,"internally_reserved":0,"allocated":156200960,"data_stored":155997298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539881167},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155156480,"data_stored":155156480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154034176,"data_stored":154034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153718784,"data_stored":153718784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":156868608,"data_stored":156868608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9121", 'reported_seq': 9353, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.794751+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:31.794751+0000', 'last_peered': '2024-04-26T02:32:31.794751+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:31.794751+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9121, 'log_dups_size': 0, 'ondisk_log_size': 9121, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19685376, 'num_objects': 2403, 'num_object_clones': 0, 'num_object_copies': 9612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9121, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 476}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9457", 'reported_seq': 9684, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:33.871662+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:33.871662+0000', 'last_peered': '2024-04-26T02:32:33.871662+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:33.871662+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9457, 'log_dups_size': 0, 'ondisk_log_size': 9457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19452428, 'num_objects': 2375, 'num_object_clones': 0, 'num_object_copies': 9500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9455, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9271", 'reported_seq': 9440, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:35.558419+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:35.558419+0000', 'last_peered': '2024-04-26T02:32:35.558419+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:35.558419+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9271, 'log_dups_size': 0, 'ondisk_log_size': 9271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19323166, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9270, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 376}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:38.011 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9178", 'reported_seq': 9409, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.564621+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:31.564621+0000', 'last_peered': '2024-04-26T02:32:31.564621+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:31.564621+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9178, 'log_dups_size': 0, 'ondisk_log_size': 9178, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19812876, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 9676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9176, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 490}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9291", 'reported_seq': 9547, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.460476+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:31.460476+0000', 'last_peered': '2024-04-26T02:32:31.460476+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:31.460476+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9291, 'log_dups_size': 0, 'ondisk_log_size': 9291, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18915328, 'num_objects': 2309, 'num_object_clones': 0, 'num_object_copies': 9236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9291, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 359}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9629", 'reported_seq': 9883, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:33.751382+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:33.751382+0000', 'last_peered': '2024-04-26T02:32:33.751382+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:33.751382+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9629, 'log_dups_size': 0, 'ondisk_log_size': 9629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19329548, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9627, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9302", 'reported_seq': 11892, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.832342+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:31.832342+0000', 'last_peered': '2024-04-26T02:32:31.832342+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:31.832342+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9302, 'log_dups_size': 0, 'ondisk_log_size': 9302, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19964952, 'num_objects': 2438, 'num_object_clones': 0, 'num_object_copies': 9752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 746, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9299, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1264, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9291", 'reported_seq': 9465, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:35.549295+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:35.549295+0000', 'last_peered': '2024-04-26T02:32:35.549295+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:35.549295+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9291, 'log_dups_size': 0, 'ondisk_log_size': 9291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18616844, 'num_objects': 2273, 'num_object_clones': 0, 'num_object_copies': 9092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9289, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 278}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9295", 'reported_seq': 9529, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.853960+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:31.853960+0000', 'last_peered': '2024-04-26T02:32:31.853960+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:31.853960+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9295, 'log_dups_size': 0, 'ondisk_log_size': 9295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19852050, 'num_objects': 2424, 'num_object_clones': 0, 'num_object_copies': 9696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9292, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 490}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9232", 'reported_seq': 9461, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.680235+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:31.680235+0000', 'last_peered': '2024-04-26T02:32:31.680235+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:31.680235+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9232, 'log_dups_size': 0, 'ondisk_log_size': 9232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19372294, 'num_objects': 2365, 'num_object_clones': 0, 'num_object_copies': 9460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9231, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9440", 'reported_seq': 9657, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:35.283605+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:35.283605+0000', 'last_peered': '2024-04-26T02:32:35.283605+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:35.283605+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9440, 'log_dups_size': 0, 'ondisk_log_size': 9440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19159302, 'num_objects': 2339, 'num_object_clones': 0, 'num_object_copies': 9356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9439, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 294}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9670", 'reported_seq': 9917, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:31.843552+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:31.843552+0000', 'last_peered': '2024-04-26T02:32:31.843552+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:31.843552+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9670, 'log_dups_size': 0, 'ondisk_log_size': 9670, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20083212, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9668, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 473}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:38.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9456", 'reported_seq': 9685, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:33.846382+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:33.846382+0000', 'last_peered': '2024-04-26T02:32:33.846382+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:33.846382+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9456, 'log_dups_size': 0, 'ondisk_log_size': 9456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19173900, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 9364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9454, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:38.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:39.732 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:39.732 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24378/150096 objects degraded (16.242%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":556,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37524,"data_bytes":307338432,"bytes_used":2659844096,"bytes_avail":383887212544,"bytes_total":386547056640,"degraded_objects":24378,"degraded_total":150096,"degraded_ratio":0.16241605372561557,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":211},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-04-26T02:31:48.648203+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:32:39.755 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:39.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:40.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:40.870 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:40.961 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-26T02:32:40.675034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":304225472,"num_objects":37144,"num_object_clones":0,"num_object_copies":148576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150413,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8391,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150440,"ondisk_log_size":150440,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2603988,"kb_used_data":596712,"kb_used_omap":14,"kb_used_meta":2007217,"kb_avail":374883372,"statfs":{"total":386547056640,"available":383880572928,"internally_reserved":0,"allocated":611033088,"data_stored":610218440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2055390771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,3,6,16,13,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20037632,"num_objects":-2446,"num_object_clones":0,"num_object_copies":-9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2446,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005874"},"pg_stats":[{"pgid":"1.f","version":"38'9191","reported_seq":9423,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.874667+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:36.874667+0000","last_peered":"2024-04-26T02:32:36.874667+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:36.874667+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9191,"log_dups_size":0,"ondisk_log_size":9191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19111936,"num_objects":2333,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9523","reported_seq":9751,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:39.008014+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:39.008014+0000","last_peered":"2024-04-26T02:32:39.008014+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:39.008014+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9523,"log_dups_size":0,"ondisk_log_size":9523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18911756,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9271","reported_seq":9440,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.558419+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:35.558419+0000","last_peered":"2024-04-26T02:32:35.558419+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:35.558419+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19323166,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9270,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9176","reported_seq":13201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.602401+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:40.602401+0000","last_peered":"2024-04-26T02:32:40.602401+0000","last_clean":"2024-04-26T02:32:40.602401+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:40.602401+0000","last_undegraded":"2024-04-26T02:32:40.602401+0000","last_fullsized":"2024-04-26T02:32:40.602401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:44.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870796,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9235","reported_seq":9466,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.858077+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:36.858077+0000","last_peered":"2024-04-26T02:32:36.858077+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:36.858077+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19345932,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9233,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9336","reported_seq":9592,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.959025+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:36.959025+0000","last_peered":"2024-04-26T02:32:36.959025+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:36.959025+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9470","reported_seq":13723,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:38.490773+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:38.490773+0000","last_peered":"2024-04-26T02:32:38.490773+0000","last_clean":"2024-04-26T02:32:38.490773+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:38.490773+0000","last_undegraded":"2024-04-26T02:32:38.490773+0000","last_fullsized":"2024-04-26T02:32:38.490773+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9470,"log_dups_size":0,"ondisk_log_size":9470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:40.014340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739724,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9692","reported_seq":9946,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:38.894087+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:38.894087+0000","last_peered":"2024-04-26T02:32:38.894087+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:38.894087+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18813452,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9360","reported_seq":12082,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.886931+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:36.886931+0000","last_peered":"2024-04-26T02:32:36.886931+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:36.886931+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19489816,"num_objects":2380,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9291","reported_seq":9465,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.549295+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:35.549295+0000","last_peered":"2024-04-26T02:32:35.549295+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:35.549295+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9291,"log_dups_size":0,"ondisk_log_size":9291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18616844,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":278}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9363","reported_seq":9597,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.945890+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:36.945890+0000","last_peered":"2024-04-26T02:32:36.945890+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:36.945890+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9363,"log_dups_size":0,"ondisk_log_size":9363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19294994,"num_objects":2356,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9310","reported_seq":9539,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.754997+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:36.754997+0000","last_peered":"2024-04-26T02:32:36.754997+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:36.754997+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18733318,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":389}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9440","reported_seq":9657,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:35.283605+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:35.283605+0000","last_peered":"2024-04-26T02:32:35.283605+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:35.283605+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19159302,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":294}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9737","reported_seq":9984,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:36.915696+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:36.915696+0000","last_peered":"2024-04-26T02:32:36.915696+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:36.915696+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9737,"log_dups_size":0,"ondisk_log_size":9737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19534348,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9735,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":406}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9537","reported_seq":13747,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.534220+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:40.534220+0000","last_peered":"2024-04-26T02:32:40.534220+0000","last_clean":"2024-04-26T02:32:40.534220+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:40.534220+0000","last_undegraded":"2024-04-26T02:32:40.534220+0000","last_fullsized":"2024-04-26T02:32:40.534220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18985484,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9508","reported_seq":9737,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:38.827430+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:38.827430+0000","last_peered":"2024-04-26T02:32:38.827430+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:38.827430+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9508,"log_dups_size":0,"ondisk_log_size":9508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18747916,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":304225472,"num_objects":37144,"num_object_clones":0,"num_object_copies":148576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150413,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8391,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":607105024,"data_stored":607105024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150440,"ondisk_log_size":150440,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117096,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673336,"kb_used_data":150192,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93698504,"statfs":{"total":96636764160,"available":95947268096,"internally_reserved":0,"allocated":153796608,"data_stored":153592946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":535686864},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1,4,4,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664900,"kb_used_data":150576,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93706940,"statfs":{"total":96636764160,"available":95955906560,"internally_reserved":0,"allocated":154189824,"data_stored":153986162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580244,"kb_used_data":147404,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93791596,"statfs":{"total":96636764160,"available":96042594304,"internally_reserved":0,"allocated":150941696,"data_stored":150738034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685508,"kb_used_data":148540,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93686332,"statfs":{"total":96636764160,"available":95934803968,"internally_reserved":0,"allocated":152104960,"data_stored":151901298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150913024,"data_stored":150913024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149893120,"data_stored":149893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153718784,"data_stored":153718784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":152580096,"data_stored":152580096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:43.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:44.874 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:44.874 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:44.929 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-04-26T02:32:44.679045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296836288,"num_objects":36242,"num_object_clones":0,"num_object_copies":144968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151315,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151342,"ondisk_log_size":151342,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643264,"kb_used_data":583216,"kb_used_omap":14,"kb_used_meta":2059953,"kb_avail":374844096,"statfs":{"total":386547056640,"available":383840354304,"internally_reserved":0,"allocated":597213184,"data_stored":596398536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2109392435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,5,11,10,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008876"},"pg_stats":[{"pgid":"1.f","version":"38'9257","reported_seq":9489,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.756144+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:41.756144+0000","last_peered":"2024-04-26T02:32:41.756144+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:41.756144+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9257,"log_dups_size":0,"ondisk_log_size":9257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571264,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9257,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9576","reported_seq":9804,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.063781+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:44.063781+0000","last_peered":"2024-04-26T02:32:44.063781+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:44.063781+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18477580,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9331","reported_seq":9500,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.330905+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:40.330905+0000","last_peered":"2024-04-26T02:32:40.330905+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:40.330905+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9331,"log_dups_size":0,"ondisk_log_size":9331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18831646,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9330,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9176","reported_seq":13201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.602401+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:40.602401+0000","last_peered":"2024-04-26T02:32:40.602401+0000","last_clean":"2024-04-26T02:32:40.602401+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:40.602401+0000","last_undegraded":"2024-04-26T02:32:40.602401+0000","last_fullsized":"2024-04-26T02:32:40.602401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:44.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870796,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9303","reported_seq":9534,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.913391+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:41.913391+0000","last_peered":"2024-04-26T02:32:41.913391+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:41.913391+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18788876,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9416","reported_seq":9672,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.052264+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:42.052264+0000","last_peered":"2024-04-26T02:32:42.052264+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:42.052264+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9529","reported_seq":13782,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:43.994743+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:43.994743+0000","last_peered":"2024-04-26T02:32:43.994743+0000","last_clean":"2024-04-26T02:32:43.994743+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:43.994743+0000","last_undegraded":"2024-04-26T02:32:43.994743+0000","last_fullsized":"2024-04-26T02:32:43.994743+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:45.043273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18256396,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9761","reported_seq":10014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:43.976647+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:43.976647+0000","last_peered":"2024-04-26T02:32:43.976647+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:43.976647+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18248204,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9429","reported_seq":12227,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.077919+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:42.077919+0000","last_peered":"2024-04-26T02:32:42.077919+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:42.077919+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18924568,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9426,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1368,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9356","reported_seq":9530,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.260845+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:40.260845+0000","last_peered":"2024-04-26T02:32:40.260845+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:40.260845+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084364,"num_objects":2208,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9416","reported_seq":9651,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.054460+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:42.054460+0000","last_peered":"2024-04-26T02:32:42.054460+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:42.054460+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18860818,"num_objects":2303,"num_object_clones":0,"num_object_copies":9212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9368","reported_seq":9597,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.038627+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:42.038627+0000","last_peered":"2024-04-26T02:32:42.038627+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:42.038627+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18258182,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9515","reported_seq":9732,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.457209+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:40.457209+0000","last_peered":"2024-04-26T02:32:40.457209+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:40.457209+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18544902,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9793","reported_seq":10040,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.789385+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:41.789385+0000","last_peered":"2024-04-26T02:32:41.789385+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:41.789385+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19075596,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9537","reported_seq":13747,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.534220+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:40.534220+0000","last_peered":"2024-04-26T02:32:40.534220+0000","last_clean":"2024-04-26T02:32:40.534220+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:40.534220+0000","last_undegraded":"2024-04-26T02:32:40.534220+0000","last_fullsized":"2024-04-26T02:32:40.534220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18985484,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9579","reported_seq":9808,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.028300+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:44.028300+0000","last_peered":"2024-04-26T02:32:44.028300+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:44.028300+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166284,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":296836288,"num_objects":36242,"num_object_clones":0,"num_object_copies":144968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151315,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":594878464,"data_stored":594878464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151342,"ondisk_log_size":151342,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694924,"kb_used_data":145384,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93676916,"statfs":{"total":96636764160,"available":95925161984,"internally_reserved":0,"allocated":148873216,"data_stored":148669554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":562687696},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683120,"kb_used_data":146600,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93688720,"statfs":{"total":96636764160,"available":95937249280,"internally_reserved":0,"allocated":150118400,"data_stored":149914738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580244,"kb_used_data":147404,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93791596,"statfs":{"total":96636764160,"available":96042594304,"internally_reserved":0,"allocated":150941696,"data_stored":150738034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684976,"kb_used_data":143828,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93686864,"statfs":{"total":96636764160,"available":95935348736,"internally_reserved":0,"allocated":147279872,"data_stored":147076210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146841600,"data_stored":146841600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149893120,"data_stored":149893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149676032,"data_stored":149676032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148467712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:44.931 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:46.436 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:46.436 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:46.466 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-04-26T02:32:44.679045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296836288,"num_objects":36242,"num_object_clones":0,"num_object_copies":144968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151315,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151342,"ondisk_log_size":151342,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643264,"kb_used_data":583216,"kb_used_omap":14,"kb_used_meta":2059953,"kb_avail":374844096,"statfs":{"total":386547056640,"available":383840354304,"internally_reserved":0,"allocated":597213184,"data_stored":596398536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2109392435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,5,11,10,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008876"},"pg_stats":[{"pgid":"1.f","version":"38'9257","reported_seq":9489,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.756144+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:41.756144+0000","last_peered":"2024-04-26T02:32:41.756144+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:41.756144+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9257,"log_dups_size":0,"ondisk_log_size":9257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571264,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9257,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9576","reported_seq":9804,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.063781+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:44.063781+0000","last_peered":"2024-04-26T02:32:44.063781+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:44.063781+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18477580,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9331","reported_seq":9500,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.330905+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:40.330905+0000","last_peered":"2024-04-26T02:32:40.330905+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:40.330905+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9331,"log_dups_size":0,"ondisk_log_size":9331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18831646,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9330,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9176","reported_seq":13201,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.602401+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:40.602401+0000","last_peered":"2024-04-26T02:32:40.602401+0000","last_clean":"2024-04-26T02:32:40.602401+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:40.602401+0000","last_undegraded":"2024-04-26T02:32:40.602401+0000","last_fullsized":"2024-04-26T02:32:40.602401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:44.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870796,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9303","reported_seq":9534,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.913391+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:41.913391+0000","last_peered":"2024-04-26T02:32:41.913391+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:41.913391+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18788876,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9416","reported_seq":9672,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.052264+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:42.052264+0000","last_peered":"2024-04-26T02:32:42.052264+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:42.052264+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9529","reported_seq":13782,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:43.994743+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:43.994743+0000","last_peered":"2024-04-26T02:32:43.994743+0000","last_clean":"2024-04-26T02:32:43.994743+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:43.994743+0000","last_undegraded":"2024-04-26T02:32:43.994743+0000","last_fullsized":"2024-04-26T02:32:43.994743+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:45.043273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18256396,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9761","reported_seq":10014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:43.976647+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:43.976647+0000","last_peered":"2024-04-26T02:32:43.976647+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:43.976647+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18248204,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9429","reported_seq":12227,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.077919+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:42.077919+0000","last_peered":"2024-04-26T02:32:42.077919+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:42.077919+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18924568,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9426,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1368,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9356","reported_seq":9530,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.260845+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:40.260845+0000","last_peered":"2024-04-26T02:32:40.260845+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:40.260845+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084364,"num_objects":2208,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":213}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9416","reported_seq":9651,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.054460+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:42.054460+0000","last_peered":"2024-04-26T02:32:42.054460+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:42.054460+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18860818,"num_objects":2303,"num_object_clones":0,"num_object_copies":9212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9413,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9368","reported_seq":9597,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:42.038627+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:42.038627+0000","last_peered":"2024-04-26T02:32:42.038627+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:42.038627+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18258182,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9515","reported_seq":9732,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:40.457209+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:40.457209+0000","last_peered":"2024-04-26T02:32:40.457209+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:40.457209+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9515,"log_dups_size":0,"ondisk_log_size":9515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18544902,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9793","reported_seq":10040,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:41.789385+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:41.789385+0000","last_peered":"2024-04-26T02:32:41.789385+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:41.789385+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19075596,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9537","reported_seq":13747,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:40.534220+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:40.534220+0000","last_peered":"2024-04-26T02:32:40.534220+0000","last_clean":"2024-04-26T02:32:40.534220+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:40.534220+0000","last_undegraded":"2024-04-26T02:32:40.534220+0000","last_fullsized":"2024-04-26T02:32:40.534220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18985484,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9579","reported_seq":9808,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.028300+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:44.028300+0000","last_peered":"2024-04-26T02:32:44.028300+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:44.028300+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166284,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":296836288,"num_objects":36242,"num_object_clones":0,"num_object_copies":144968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151315,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":594878464,"data_stored":594878464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151342,"ondisk_log_size":151342,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694924,"kb_used_data":145384,"kb_used_omap":4,"kb_used_meta":549499,"kb_avail":93676916,"statfs":{"total":96636764160,"available":95925161984,"internally_reserved":0,"allocated":148873216,"data_stored":148669554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":562687696},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683120,"kb_used_data":146600,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93688720,"statfs":{"total":96636764160,"available":95937249280,"internally_reserved":0,"allocated":150118400,"data_stored":149914738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580244,"kb_used_data":147404,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93791596,"statfs":{"total":96636764160,"available":96042594304,"internally_reserved":0,"allocated":150941696,"data_stored":150738034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684976,"kb_used_data":143828,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93686864,"statfs":{"total":96636764160,"available":95935348736,"internally_reserved":0,"allocated":147279872,"data_stored":147076210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146841600,"data_stored":146841600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149893120,"data_stored":149893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149676032,"data_stored":149676032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148467712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9257", 'reported_seq': 9489, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:41.756144+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:41.756144+0000', 'last_peered': '2024-04-26T02:32:41.756144+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:41.756144+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9257, 'log_dups_size': 0, 'ondisk_log_size': 9257, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18571264, 'num_objects': 2267, 'num_object_clones': 0, 'num_object_copies': 9068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9257, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9576", 'reported_seq': 9804, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:44.063781+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:44.063781+0000', 'last_peered': '2024-04-26T02:32:44.063781+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:44.063781+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9576, 'log_dups_size': 0, 'ondisk_log_size': 9576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18477580, 'num_objects': 2256, 'num_object_clones': 0, 'num_object_copies': 9024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9574, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9331", 'reported_seq': 9500, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:40.330905+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:40.330905+0000', 'last_peered': '2024-04-26T02:32:40.330905+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:40.330905+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9331, 'log_dups_size': 0, 'ondisk_log_size': 9331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18831646, 'num_objects': 2300, 'num_object_clones': 0, 'num_object_copies': 9200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2300, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9330, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 316}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9303", 'reported_seq': 9534, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:41.913391+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:41.913391+0000', 'last_peered': '2024-04-26T02:32:41.913391+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:41.913391+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9303, 'log_dups_size': 0, 'ondisk_log_size': 9303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18788876, 'num_objects': 2294, 'num_object_clones': 0, 'num_object_copies': 9176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9301, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9416", 'reported_seq': 9672, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:42.052264+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:42.052264+0000', 'last_peered': '2024-04-26T02:32:42.052264+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:42.052264+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9416, 'log_dups_size': 0, 'ondisk_log_size': 9416, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17891328, 'num_objects': 2184, 'num_object_clones': 0, 'num_object_copies': 8736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9416, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9761", 'reported_seq': 10014, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:43.976647+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:43.976647+0000', 'last_peered': '2024-04-26T02:32:43.976647+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:43.976647+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9761, 'log_dups_size': 0, 'ondisk_log_size': 9761, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18248204, 'num_objects': 2228, 'num_object_clones': 0, 'num_object_copies': 8912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9759, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9429", 'reported_seq': 12227, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:42.077919+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:42.077919+0000', 'last_peered': '2024-04-26T02:32:42.077919+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:42.077919+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9429, 'log_dups_size': 0, 'ondisk_log_size': 9429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18924568, 'num_objects': 2311, 'num_object_clones': 0, 'num_object_copies': 9244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 642, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9426, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1368, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1669}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9356", 'reported_seq': 9530, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:40.260845+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:40.260845+0000', 'last_peered': '2024-04-26T02:32:40.260845+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:40.260845+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9356, 'log_dups_size': 0, 'ondisk_log_size': 9356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18084364, 'num_objects': 2208, 'num_object_clones': 0, 'num_object_copies': 8832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9354, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 213}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9416", 'reported_seq': 9651, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:42.054460+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:42.054460+0000', 'last_peered': '2024-04-26T02:32:42.054460+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:42.054460+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9416, 'log_dups_size': 0, 'ondisk_log_size': 9416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18860818, 'num_objects': 2303, 'num_object_clones': 0, 'num_object_copies': 9212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9413, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 369}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:46.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9368", 'reported_seq': 9597, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:42.038627+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:42.038627+0000', 'last_peered': '2024-04-26T02:32:42.038627+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:42.038627+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9368, 'log_dups_size': 0, 'ondisk_log_size': 9368, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18258182, 'num_objects': 2229, 'num_object_clones': 0, 'num_object_copies': 8916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9367, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9515", 'reported_seq': 9732, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:40.457209+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:40.457209+0000', 'last_peered': '2024-04-26T02:32:40.457209+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:40.457209+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9515, 'log_dups_size': 0, 'ondisk_log_size': 9515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18544902, 'num_objects': 2264, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9514, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9793", 'reported_seq': 10040, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:41.789385+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:41.789385+0000', 'last_peered': '2024-04-26T02:32:41.789385+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:41.789385+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9793, 'log_dups_size': 0, 'ondisk_log_size': 9793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19075596, 'num_objects': 2329, 'num_object_clones': 0, 'num_object_copies': 9316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9791, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:46.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9579", 'reported_seq': 9808, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:44.028300+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:44.028300+0000', 'last_peered': '2024-04-26T02:32:44.028300+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:44.028300+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9579, 'log_dups_size': 0, 'ondisk_log_size': 9579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18166284, 'num_objects': 2218, 'num_object_clones': 0, 'num_object_copies': 8872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9577, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:46.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:48.024 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24340/143788 objects degraded (16.928%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":564,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35947,"data_bytes":294419648,"bytes_used":2706620416,"bytes_avail":383840436224,"bytes_total":386547056640,"degraded_objects":24340,"degraded_total":143788,"degraded_ratio":0.16927699112582414,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:32:48.155 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:48.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:49.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:49.871 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:49.955 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-04-26T02:32:48.682545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291028160,"num_objects":35533,"num_object_clones":0,"num_object_copies":142132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35533,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152024,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8550,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152051,"ondisk_log_size":152051,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626936,"kb_used_data":571060,"kb_used_omap":14,"kb_used_meta":2055793,"kb_avail":374860424,"statfs":{"total":386547056640,"available":383857074176,"internally_reserved":0,"allocated":584765440,"data_stored":583950792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2105132595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,9,11,7,4,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19939328,"num_objects":-2434,"num_object_clones":0,"num_object_copies":-9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011381"},"pg_stats":[{"pgid":"1.f","version":"38'9320","reported_seq":9552,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.130012+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:47.130012+0000","last_peered":"2024-04-26T02:32:47.130012+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:47.130012+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18055168,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9576","reported_seq":9804,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.063781+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:44.063781+0000","last_peered":"2024-04-26T02:32:44.063781+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:44.063781+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18477580,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9386","reported_seq":9555,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:45.721725+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:45.721725+0000","last_peered":"2024-04-26T02:32:45.721725+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:45.721725+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18381086,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9385,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9231","reported_seq":13257,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:45.472336+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:45.472336+0000","last_peered":"2024-04-26T02:32:45.472336+0000","last_clean":"2024-04-26T02:32:45.472336+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:45.472336+0000","last_undegraded":"2024-04-26T02:32:45.472336+0000","last_fullsized":"2024-04-26T02:32:45.472336+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:49.639576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18420236,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9229,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9343","reported_seq":9573,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:46.750623+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:46.750623+0000","last_peered":"2024-04-26T02:32:46.750623+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:46.750623+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461196,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9341,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9479","reported_seq":9735,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.142065+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:47.142065+0000","last_peered":"2024-04-26T02:32:47.142065+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:47.142065+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9479,"log_dups_size":0,"ondisk_log_size":9479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17375232,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9479,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9529","reported_seq":13782,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:43.994743+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:43.994743+0000","last_peered":"2024-04-26T02:32:43.994743+0000","last_clean":"2024-04-26T02:32:43.994743+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:43.994743+0000","last_undegraded":"2024-04-26T02:32:43.994743+0000","last_fullsized":"2024-04-26T02:32:43.994743+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:45.043273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18256396,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9761","reported_seq":10014,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:43.976647+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:43.976647+0000","last_peered":"2024-04-26T02:32:43.976647+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:43.976647+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18248204,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9481","reported_seq":12521,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.157804+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:47.157804+0000","last_peered":"2024-04-26T02:32:47.157804+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:47.157804+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18498584,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1489,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9423","reported_seq":9595,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:45.442501+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:45.442501+0000","last_peered":"2024-04-26T02:32:45.442501+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:45.442501+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17535500,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9421,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9479","reported_seq":9714,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.129197+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:47.129197+0000","last_peered":"2024-04-26T02:32:47.129197+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:47.129197+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9479,"log_dups_size":0,"ondisk_log_size":9479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18344722,"num_objects":2240,"num_object_clones":0,"num_object_copies":8960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9476,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9424","reported_seq":9653,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.107121+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:47.107121+0000","last_peered":"2024-04-26T02:32:47.107121+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:47.107121+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17799430,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9569","reported_seq":9787,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:45.684998+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:45.684998+0000","last_peered":"2024-04-26T02:32:45.684998+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:45.684998+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9569,"log_dups_size":0,"ondisk_log_size":9569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18102534,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":165}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9870","reported_seq":10117,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:47.094657+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:47.094657+0000","last_peered":"2024-04-26T02:32:47.094657+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:47.094657+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9870,"log_dups_size":0,"ondisk_log_size":9870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18444812,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9868,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9601","reported_seq":13811,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:45.685140+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:45.685140+0000","last_peered":"2024-04-26T02:32:45.685140+0000","last_clean":"2024-04-26T02:32:45.685140+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:45.685140+0000","last_undegraded":"2024-04-26T02:32:45.685140+0000","last_fullsized":"2024-04-26T02:32:45.685140+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18461196,"num_objects":2254,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9599,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9579","reported_seq":9808,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:44.028300+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:44.028300+0000","last_peered":"2024-04-26T02:32:44.028300+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:44.028300+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9579,"log_dups_size":0,"ondisk_log_size":9579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18166284,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291028160,"num_objects":35533,"num_object_clones":0,"num_object_copies":142132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35533,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152024,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8550,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":582881280,"data_stored":582881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152051,"ondisk_log_size":152051,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117098,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678676,"kb_used_data":141644,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93693164,"statfs":{"total":96636764160,"available":95941799936,"internally_reserved":0,"allocated":145043456,"data_stored":144839794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":549908176},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,2,3,0,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683436,"kb_used_data":142740,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93688404,"statfs":{"total":96636764160,"available":95936925696,"internally_reserved":0,"allocated":146165760,"data_stored":145962098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579848,"kb_used_data":142848,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93791992,"statfs":{"total":96636764160,"available":96042999808,"internally_reserved":0,"allocated":146276352,"data_stored":146072690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684976,"kb_used_data":143828,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93686864,"statfs":{"total":96636764160,"available":95935348736,"internally_reserved":0,"allocated":147279872,"data_stored":147076210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":146841600,"data_stored":146841600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145854464,"data_stored":145854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145629184,"data_stored":145629184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144556032,"data_stored":144556032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:52.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:54.529 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:54.649 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-04-26T02:32:52.683475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283032768,"num_objects":34557,"num_object_clones":0,"num_object_copies":138228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153000,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153027,"ondisk_log_size":153027,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2633820,"kb_used_data":555768,"kb_used_omap":14,"kb_used_meta":2078001,"kb_avail":374853540,"statfs":{"total":386547056640,"available":383850024960,"internally_reserved":0,"allocated":569106432,"data_stored":568291784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2127873587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,8,11,9,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21192704,"num_objects":-2587,"num_object_clones":0,"num_object_copies":-10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008441"},"pg_stats":[{"pgid":"1.f","version":"38'9376","reported_seq":9608,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.220932+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:52.220932+0000","last_peered":"2024-04-26T02:32:52.220932+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:52.220932+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17596416,"num_objects":2148,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9376,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9623","reported_seq":9851,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:48.870261+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:48.870261+0000","last_peered":"2024-04-26T02:32:48.870261+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:48.870261+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18092556,"num_objects":2209,"num_object_clones":0,"num_object_copies":8836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9621,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9452","reported_seq":9622,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.774449+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:50.774449+0000","last_peered":"2024-04-26T02:32:50.774449+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:50.774449+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17840414,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9451,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9287","reported_seq":13313,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:50.622646+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:50.622646+0000","last_peered":"2024-04-26T02:32:50.622646+0000","last_clean":"2024-04-26T02:32:50.622646+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:50.622646+0000","last_undegraded":"2024-04-26T02:32:50.622646+0000","last_fullsized":"2024-04-26T02:32:50.622646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:55.605098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17961484,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9411","reported_seq":9642,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.153976+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:52.153976+0000","last_peered":"2024-04-26T02:32:52.153976+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:52.153976+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17904140,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9409,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9527","reported_seq":9783,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.147014+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:52.147014+0000","last_peered":"2024-04-26T02:32:52.147014+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:52.147014+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9527,"log_dups_size":0,"ondisk_log_size":9527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16982016,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9578","reported_seq":13831,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:48.807182+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:48.807182+0000","last_peered":"2024-04-26T02:32:48.807182+0000","last_clean":"2024-04-26T02:32:48.807182+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:48.807182+0000","last_undegraded":"2024-04-26T02:32:48.807182+0000","last_fullsized":"2024-04-26T02:32:48.807182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:51.033899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17854988,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9825","reported_seq":10080,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:49.076963+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:49.076963+0000","last_peered":"2024-04-26T02:32:49.076963+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:49.076963+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9825,"log_dups_size":0,"ondisk_log_size":9825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17723916,"num_objects":2164,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9545","reported_seq":12729,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.187772+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:52.187772+0000","last_peered":"2024-04-26T02:32:52.187772+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:52.187772+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17974296,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1561,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9499","reported_seq":9673,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.802378+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:50.802378+0000","last_peered":"2024-04-26T02:32:50.802378+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:50.802378+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16912908,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9537","reported_seq":9772,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.300384+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:52.300384+0000","last_peered":"2024-04-26T02:32:52.300384+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:52.300384+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17869586,"num_objects":2182,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9497","reported_seq":9726,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.236128+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:52.236128+0000","last_peered":"2024-04-26T02:32:52.236128+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:52.236128+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17201414,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9629","reported_seq":9847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.667103+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:50.667103+0000","last_peered":"2024-04-26T02:32:50.667103+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:50.667103+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17611014,"num_objects":2150,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9935","reported_seq":10181,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.119224+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:52.119224+0000","last_peered":"2024-04-26T02:32:52.119224+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:52.119224+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17912332,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9668","reported_seq":13878,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:50.568991+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:50.568991+0000","last_peered":"2024-04-26T02:32:50.568991+0000","last_clean":"2024-04-26T02:32:50.568991+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:50.568991+0000","last_undegraded":"2024-04-26T02:32:50.568991+0000","last_fullsized":"2024-04-26T02:32:50.568991+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:51.654753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17912332,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9638","reported_seq":9867,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:48.883123+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:48.883123+0000","last_peered":"2024-04-26T02:32:48.883123+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:48.883123+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17682956,"num_objects":2159,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":283032768,"num_objects":34557,"num_object_clones":0,"num_object_copies":138228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153000,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":566808576,"data_stored":566808576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153027,"ondisk_log_size":153027,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696832,"kb_used_data":137588,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93675008,"statfs":{"total":96636764160,"available":95923208192,"internally_reserved":0,"allocated":140890112,"data_stored":140686450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683848,"kb_used_data":139008,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93687992,"statfs":{"total":96636764160,"available":95936503808,"internally_reserved":0,"allocated":142344192,"data_stored":142140530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567724,"kb_used_data":139044,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93804116,"statfs":{"total":96636764160,"available":96055414784,"internally_reserved":0,"allocated":142381056,"data_stored":142177394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685416,"kb_used_data":140128,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93686424,"statfs":{"total":96636764160,"available":95934898176,"internally_reserved":0,"allocated":143491072,"data_stored":143287410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142979072,"data_stored":142979072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141922304,"data_stored":141922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141574144,"data_stored":141574144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140333056,"data_stored":140333056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:54.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:55.828 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:55.829 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:55.856 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-04-26T02:32:54.686542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280689856,"num_objects":34271,"num_object_clones":0,"num_object_copies":137084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153286,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153313,"ondisk_log_size":153313,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2617196,"kb_used_data":551612,"kb_used_omap":14,"kb_used_meta":2065521,"kb_avail":374870164,"statfs":{"total":386547056640,"available":383867047936,"internally_reserved":0,"allocated":564850688,"data_stored":564036040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2115094067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,8,13,11,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18210816,"num_objects":-2223,"num_object_clones":0,"num_object_copies":-8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008003"},"pg_stats":[{"pgid":"1.f","version":"38'9376","reported_seq":9608,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.220932+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:52.220932+0000","last_peered":"2024-04-26T02:32:52.220932+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:52.220932+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17596416,"num_objects":2148,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9376,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9696","reported_seq":9924,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:54.092676+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:54.092676+0000","last_peered":"2024-04-26T02:32:54.092676+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:54.092676+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17494540,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9452","reported_seq":9622,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.774449+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:50.774449+0000","last_peered":"2024-04-26T02:32:50.774449+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:50.774449+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17840414,"num_objects":2179,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9451,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9287","reported_seq":13313,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:50.622646+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:50.622646+0000","last_peered":"2024-04-26T02:32:50.622646+0000","last_clean":"2024-04-26T02:32:50.622646+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:50.622646+0000","last_undegraded":"2024-04-26T02:32:50.622646+0000","last_fullsized":"2024-04-26T02:32:50.622646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:55.605098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17961484,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9411","reported_seq":9642,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.153976+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:52.153976+0000","last_peered":"2024-04-26T02:32:52.153976+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:52.153976+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17904140,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9409,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9527","reported_seq":9783,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.147014+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:52.147014+0000","last_peered":"2024-04-26T02:32:52.147014+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:52.147014+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9527,"log_dups_size":0,"ondisk_log_size":9527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16982016,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9656","reported_seq":13909,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:53.992494+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:53.992494+0000","last_peered":"2024-04-26T02:32:53.992494+0000","last_clean":"2024-04-26T02:32:53.992494+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:53.992494+0000","last_undegraded":"2024-04-26T02:32:53.992494+0000","last_fullsized":"2024-04-26T02:32:53.992494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:56.969386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216012,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9894","reported_seq":10149,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:54.070069+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:54.070069+0000","last_peered":"2024-04-26T02:32:54.070069+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:54.070069+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17158668,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":43}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9545","reported_seq":12729,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.187772+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:52.187772+0000","last_peered":"2024-04-26T02:32:52.187772+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:52.187772+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17974296,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1561,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9499","reported_seq":9673,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.802378+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:50.802378+0000","last_peered":"2024-04-26T02:32:50.802378+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:50.802378+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16912908,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9537","reported_seq":9772,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.300384+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:52.300384+0000","last_peered":"2024-04-26T02:32:52.300384+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:52.300384+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17869586,"num_objects":2182,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9497","reported_seq":9726,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.236128+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:52.236128+0000","last_peered":"2024-04-26T02:32:52.236128+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:52.236128+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17201414,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9629","reported_seq":9847,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:50.667103+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:50.667103+0000","last_peered":"2024-04-26T02:32:50.667103+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:50.667103+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17611014,"num_objects":2150,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9935","reported_seq":10181,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:52.119224+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:52.119224+0000","last_peered":"2024-04-26T02:32:52.119224+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:52.119224+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17912332,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9668","reported_seq":13878,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:50.568991+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:50.568991+0000","last_peered":"2024-04-26T02:32:50.568991+0000","last_clean":"2024-04-26T02:32:50.568991+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:50.568991+0000","last_undegraded":"2024-04-26T02:32:50.568991+0000","last_fullsized":"2024-04-26T02:32:50.568991+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9668,"log_dups_size":0,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:51.654753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17912332,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9704","reported_seq":9931,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:53.827233+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:53.827233+0000","last_peered":"2024-04-26T02:32:53.827233+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:53.827233+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17142284,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280689856,"num_objects":34271,"num_object_clones":0,"num_object_copies":137084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153286,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8622,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":562610176,"data_stored":562610176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153313,"ondisk_log_size":153313,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696832,"kb_used_data":137588,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93675008,"statfs":{"total":96636764160,"available":95923208192,"internally_reserved":0,"allocated":140890112,"data_stored":140686450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683848,"kb_used_data":139008,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93687992,"statfs":{"total":96636764160,"available":95936503808,"internally_reserved":0,"allocated":142344192,"data_stored":142140530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567724,"kb_used_data":139044,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93804116,"statfs":{"total":96636764160,"available":96055414784,"internally_reserved":0,"allocated":142381056,"data_stored":142177394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668792,"kb_used_data":135972,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93703048,"statfs":{"total":96636764160,"available":95951921152,"internally_reserved":0,"allocated":139235328,"data_stored":139031666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":138780672,"data_stored":138780672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141922304,"data_stored":141922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141574144,"data_stored":141574144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140333056,"data_stored":140333056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:32:55.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:32:55.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9376", 'reported_seq': 9608, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.220932+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:32:52.220932+0000', 'last_peered': '2024-04-26T02:32:52.220932+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:32:52.220932+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9376, 'log_dups_size': 0, 'ondisk_log_size': 9376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17596416, 'num_objects': 2148, 'num_object_clones': 0, 'num_object_copies': 8592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9376, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 221}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9696", 'reported_seq': 9924, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:54.092676+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:32:54.092676+0000', 'last_peered': '2024-04-26T02:32:54.092676+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:32:54.092676+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9696, 'log_dups_size': 0, 'ondisk_log_size': 9696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17494540, 'num_objects': 2136, 'num_object_clones': 0, 'num_object_copies': 8544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9694, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9452", 'reported_seq': 9622, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:50.774449+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:32:50.774449+0000', 'last_peered': '2024-04-26T02:32:50.774449+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:32:50.774449+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9452, 'log_dups_size': 0, 'ondisk_log_size': 9452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17840414, 'num_objects': 2179, 'num_object_clones': 0, 'num_object_copies': 8716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2179, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9451, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 195}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9411", 'reported_seq': 9642, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.153976+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:32:52.153976+0000', 'last_peered': '2024-04-26T02:32:52.153976+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:32:52.153976+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9411, 'log_dups_size': 0, 'ondisk_log_size': 9411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17904140, 'num_objects': 2186, 'num_object_clones': 0, 'num_object_copies': 8744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9409, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9527", 'reported_seq': 9783, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.147014+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:32:52.147014+0000', 'last_peered': '2024-04-26T02:32:52.147014+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:32:52.147014+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9527, 'log_dups_size': 0, 'ondisk_log_size': 9527, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16982016, 'num_objects': 2073, 'num_object_clones': 0, 'num_object_copies': 8292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9527, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.858 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'9894", 'reported_seq': 10149, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:54.070069+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:32:54.070069+0000', 'last_peered': '2024-04-26T02:32:54.070069+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:32:54.070069+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9894, 'log_dups_size': 0, 'ondisk_log_size': 9894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17158668, 'num_objects': 2095, 'num_object_clones': 0, 'num_object_copies': 8380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9892, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 43}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9545", 'reported_seq': 12729, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.187772+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:32:52.187772+0000', 'last_peered': '2024-04-26T02:32:52.187772+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:32:52.187772+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9545, 'log_dups_size': 0, 'ondisk_log_size': 9545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17974296, 'num_objects': 2195, 'num_object_clones': 0, 'num_object_copies': 8780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 449, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9542, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1561, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1746}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9499", 'reported_seq': 9673, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:50.802378+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:32:50.802378+0000', 'last_peered': '2024-04-26T02:32:50.802378+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:32:50.802378+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9499, 'log_dups_size': 0, 'ondisk_log_size': 9499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16912908, 'num_objects': 2065, 'num_object_clones': 0, 'num_object_copies': 8260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9497, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 70}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9537", 'reported_seq': 9772, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.300384+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:32:52.300384+0000', 'last_peered': '2024-04-26T02:32:52.300384+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:32:52.300384+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9537, 'log_dups_size': 0, 'ondisk_log_size': 9537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17869586, 'num_objects': 2182, 'num_object_clones': 0, 'num_object_copies': 8728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9534, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 248}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:32:55.859 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9497", 'reported_seq': 9726, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.236128+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:32:52.236128+0000', 'last_peered': '2024-04-26T02:32:52.236128+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:32:52.236128+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9497, 'log_dups_size': 0, 'ondisk_log_size': 9497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17201414, 'num_objects': 2100, 'num_object_clones': 0, 'num_object_copies': 8400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9496, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9629", 'reported_seq': 9847, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:50.667103+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:32:50.667103+0000', 'last_peered': '2024-04-26T02:32:50.667103+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:32:50.667103+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9629, 'log_dups_size': 0, 'ondisk_log_size': 9629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17611014, 'num_objects': 2150, 'num_object_clones': 0, 'num_object_copies': 8600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9628, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'9935", 'reported_seq': 10181, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:52.119224+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:32:52.119224+0000', 'last_peered': '2024-04-26T02:32:52.119224+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:32:52.119224+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9935, 'log_dups_size': 0, 'ondisk_log_size': 9935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17912332, 'num_objects': 2187, 'num_object_clones': 0, 'num_object_copies': 8748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9933, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:32:55.860 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9704", 'reported_seq': 9931, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:32:53.827233+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:32:53.827233+0000', 'last_peered': '2024-04-26T02:32:53.827233+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:32:53.827233+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9704, 'log_dups_size': 0, 'ondisk_log_size': 9704, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17142284, 'num_objects': 2093, 'num_object_clones': 0, 'num_object_copies': 8372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9702, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:32:55.860 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:32:57.853 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:57.854 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24147/135828 objects degraded (17.778%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":574,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33957,"data_bytes":278117568,"bytes_used":2681565184,"bytes_avail":383865491456,"bytes_total":386547056640,"degraded_objects":24147,"degraded_total":135828,"degraded_ratio":0.17777630532732572,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:32:57.985 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:32:57.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:32:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:32:59.464 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:32:59.522 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-04-26T02:32:58.690008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274619584,"num_objects":33530,"num_object_clones":0,"num_object_copies":134120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154027,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8678,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154054,"ondisk_log_size":154054,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618884,"kb_used_data":539392,"kb_used_omap":14,"kb_used_meta":2079409,"kb_avail":374868476,"statfs":{"total":386547056640,"available":383865319424,"internally_reserved":0,"allocated":552337408,"data_stored":551522760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2129315379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,2,10,12,10,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19800064,"num_objects":-2417,"num_object_clones":0,"num_object_copies":-9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010534"},"pg_stats":[{"pgid":"1.f","version":"38'9428","reported_seq":9660,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.204785+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:32:57.204785+0000","last_peered":"2024-04-26T02:32:57.204785+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:32:57.204785+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":2096,"num_object_clones":0,"num_object_copies":8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9696","reported_seq":9924,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:54.092676+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:54.092676+0000","last_peered":"2024-04-26T02:32:54.092676+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:54.092676+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17494540,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":136}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9517","reported_seq":9686,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:55.930933+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:32:55.930933+0000","last_peered":"2024-04-26T02:32:55.930933+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:32:55.930933+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17307934,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9516,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":130}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9351","reported_seq":13377,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:55.694645+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:32:55.694645+0000","last_peered":"2024-04-26T02:32:55.694645+0000","last_clean":"2024-04-26T02:32:55.694645+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:32:55.694645+0000","last_undegraded":"2024-04-26T02:32:55.694645+0000","last_fullsized":"2024-04-26T02:32:55.694645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9351,"log_dups_size":0,"ondisk_log_size":9351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:00.642528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17437196,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9466","reported_seq":9697,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.063794+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:32:57.063794+0000","last_peered":"2024-04-26T02:32:57.063794+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:32:57.063794+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9466,"log_dups_size":0,"ondisk_log_size":9466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17453580,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9597","reported_seq":9853,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.256740+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:32:57.256740+0000","last_peered":"2024-04-26T02:32:57.256740+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:32:57.256740+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9597,"log_dups_size":0,"ondisk_log_size":9597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16408576,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":53}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9656","reported_seq":13909,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:53.992494+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:53.992494+0000","last_peered":"2024-04-26T02:32:53.992494+0000","last_clean":"2024-04-26T02:32:53.992494+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:53.992494+0000","last_undegraded":"2024-04-26T02:32:53.992494+0000","last_fullsized":"2024-04-26T02:32:53.992494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:56.969386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17216012,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9894","reported_seq":10149,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:54.070069+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:54.070069+0000","last_peered":"2024-04-26T02:32:54.070069+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:54.070069+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17158668,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":43}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9604","reported_seq":12901,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.268328+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:32:57.268328+0000","last_peered":"2024-04-26T02:32:57.268328+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:32:57.268328+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9604,"log_dups_size":0,"ondisk_log_size":9604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17490968,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9566","reported_seq":9740,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:55.879517+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:32:55.879517+0000","last_peered":"2024-04-26T02:32:55.879517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:32:55.879517+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16364044,"num_objects":1998,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9621","reported_seq":9854,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.316934+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:32:57.316934+0000","last_peered":"2024-04-26T02:32:57.316934+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:32:57.316934+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17181458,"num_objects":2098,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9618,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9549","reported_seq":9779,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.023416+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:32:57.023416+0000","last_peered":"2024-04-26T02:32:57.023416+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:32:57.023416+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16775430,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9548,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9682","reported_seq":9900,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:55.974314+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:32:55.974314+0000","last_peered":"2024-04-26T02:32:55.974314+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:32:55.974314+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17176838,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":52}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'9990","reported_seq":10237,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:57.144002+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:32:57.144002+0000","last_peered":"2024-04-26T02:32:57.144002+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:32:57.144002+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17461772,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9988,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9733","reported_seq":13943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:55.648510+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:32:55.648510+0000","last_peered":"2024-04-26T02:32:55.648510+0000","last_clean":"2024-04-26T02:32:55.648510+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:32:55.648510+0000","last_undegraded":"2024-04-26T02:32:55.648510+0000","last_fullsized":"2024-04-26T02:32:55.648510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9733,"log_dups_size":0,"ondisk_log_size":9733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:57.643357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379852,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9704","reported_seq":9931,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:53.827233+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:53.827233+0000","last_peered":"2024-04-26T02:32:53.827233+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:53.827233+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17142284,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274619584,"num_objects":33530,"num_object_clones":0,"num_object_copies":134120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154027,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8678,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":550002688,"data_stored":550002688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154054,"ondisk_log_size":154054,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117100,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697000,"kb_used_data":133588,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93674840,"statfs":{"total":96636764160,"available":95923036160,"internally_reserved":0,"allocated":136794112,"data_stored":136590450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":576909008},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,4,3,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667256,"kb_used_data":134896,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93704584,"statfs":{"total":96636764160,"available":95953494016,"internally_reserved":0,"allocated":138133504,"data_stored":137929842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585836,"kb_used_data":134936,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93786004,"statfs":{"total":96636764160,"available":96036868096,"internally_reserved":0,"allocated":138174464,"data_stored":137970802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":461696719},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668792,"kb_used_data":135972,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93703048,"statfs":{"total":96636764160,"available":95951921152,"internally_reserved":0,"allocated":139235328,"data_stored":139031666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":138780672,"data_stored":138780672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137711616,"data_stored":137711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137269248,"data_stored":137269248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":136241152,"data_stored":136241152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:02.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:04.497 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:04.498 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:04.571 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-04-26T02:33:02.691000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266542272,"num_objects":32544,"num_object_clones":0,"num_object_copies":130176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155013,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8782,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155040,"ondisk_log_size":155040,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657652,"kb_used_data":524040,"kb_used_omap":14,"kb_used_meta":2133553,"kb_avail":374829708,"statfs":{"total":386547056640,"available":383825620992,"internally_reserved":0,"allocated":536616960,"data_stored":535802312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2184758835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,9,12,16,13,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-2770,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008000"},"pg_stats":[{"pgid":"1.f","version":"38'9478","reported_seq":9710,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.887389+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:01.887389+0000","last_peered":"2024-04-26T02:33:01.887389+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:01.887389+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9478,"log_dups_size":0,"ondisk_log_size":9478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16760832,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9756","reported_seq":9981,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:59.291088+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:32:59.291088+0000","last_peered":"2024-04-26T02:32:59.291088+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:32:59.291088+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17003020,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":76}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9580","reported_seq":9750,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.064958+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:01.064958+0000","last_peered":"2024-04-26T02:33:01.064958+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:01.064958+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16791838,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9579,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":67}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9420","reported_seq":13446,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:00.803235+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:00.803235+0000","last_peered":"2024-04-26T02:33:00.803235+0000","last_clean":"2024-04-26T02:33:00.803235+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:00.803235+0000","last_undegraded":"2024-04-26T02:33:00.803235+0000","last_fullsized":"2024-04-26T02:33:00.803235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9420,"log_dups_size":0,"ondisk_log_size":9420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871948,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9529","reported_seq":9760,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.060747+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:02.060747+0000","last_peered":"2024-04-26T02:33:02.060747+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:02.060747+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937484,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9667","reported_seq":9923,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.038346+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:02.038346+0000","last_peered":"2024-04-26T02:33:02.038346+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:02.038346+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9719","reported_seq":13972,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:32:59.081176+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:32:59.081176+0000","last_peered":"2024-04-26T02:32:59.081176+0000","last_clean":"2024-04-26T02:32:59.081176+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:32:59.081176+0000","last_undegraded":"2024-04-26T02:32:59.081176+0000","last_fullsized":"2024-04-26T02:32:59.081176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9719,"log_dups_size":0,"ondisk_log_size":9719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:01.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16699916,"num_objects":2039,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9717,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'9958","reported_seq":10213,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:59.225816+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:32:59.225816+0000","last_peered":"2024-04-26T02:32:59.225816+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:32:59.225816+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9958,"log_dups_size":0,"ondisk_log_size":9958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16634380,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-21}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9679","reported_seq":13184,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.372670+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:02.372670+0000","last_peered":"2024-04-26T02:33:02.372670+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:02.372670+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16876568,"num_objects":2061,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1721,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9614","reported_seq":9788,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.057160+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:01.057160+0000","last_peered":"2024-04-26T02:33:01.057160+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:01.057160+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15970828,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-45}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9694","reported_seq":9929,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.080834+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:02.080834+0000","last_peered":"2024-04-26T02:33:02.080834+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:02.080834+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16583442,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9602","reported_seq":9832,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.010021+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:02.010021+0000","last_peered":"2024-04-26T02:33:02.010021+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:02.010021+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16341254,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":97}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9738","reported_seq":9956,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.090260+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:01.090260+0000","last_peered":"2024-04-26T02:33:01.090260+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:01.090260+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9738,"log_dups_size":0,"ondisk_log_size":9738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16718086,"num_objects":2041,"num_object_clones":0,"num_object_copies":8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9737,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-4}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10044","reported_seq":10291,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.056402+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:02.056402+0000","last_peered":"2024-04-26T02:33:02.056402+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:02.056402+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17019404,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9793","reported_seq":14003,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:00.519838+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:00.519838+0000","last_peered":"2024-04-26T02:33:00.519838+0000","last_clean":"2024-04-26T02:33:00.519838+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:00.519838+0000","last_undegraded":"2024-04-26T02:33:00.519838+0000","last_fullsized":"2024-04-26T02:33:00.519838+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:03.609797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16888332,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9769","reported_seq":9998,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:32:59.080476+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:32:59.080476+0000","last_peered":"2024-04-26T02:32:59.080476+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:32:59.080476+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16609804,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":41}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266542272,"num_objects":32544,"num_object_clones":0,"num_object_copies":130176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155013,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8782,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":534183936,"data_stored":534183936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155040,"ondisk_log_size":155040,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680624,"kb_used_data":129704,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93691216,"statfs":{"total":96636764160,"available":95939805184,"internally_reserved":0,"allocated":132816896,"data_stored":132613234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":564129488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,2,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703736,"kb_used_data":131108,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93668104,"statfs":{"total":96636764160,"available":95916138496,"internally_reserved":0,"allocated":134254592,"data_stored":134050930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586160,"kb_used_data":131112,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93785680,"statfs":{"total":96636764160,"available":96036536320,"internally_reserved":0,"allocated":134258688,"data_stored":134055026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,4,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687132,"kb_used_data":132116,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93684708,"statfs":{"total":96636764160,"available":95933140992,"internally_reserved":0,"allocated":135286784,"data_stored":135083122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134709248,"data_stored":134709248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133763072,"data_stored":133763072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133287936,"data_stored":133287936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132423680,"data_stored":132423680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:04.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:06.315 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:06.316 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:06.380 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-26T02:33:04.691505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264789184,"num_objects":32330,"num_object_clones":0,"num_object_copies":129320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155227,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8782,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155254,"ondisk_log_size":155254,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694212,"kb_used_data":520344,"kb_used_omap":14,"kb_used_meta":2173809,"kb_avail":374793148,"statfs":{"total":386547056640,"available":383788183552,"internally_reserved":0,"allocated":532832256,"data_stored":532017608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2225980979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,10,10,14,12,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18243584,"num_objects":-2227,"num_object_clones":0,"num_object_copies":-8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008030"},"pg_stats":[{"pgid":"1.f","version":"38'9478","reported_seq":9710,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.887389+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:01.887389+0000","last_peered":"2024-04-26T02:33:01.887389+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:01.887389+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9478,"log_dups_size":0,"ondisk_log_size":9478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16760832,"num_objects":2046,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9826","reported_seq":10054,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:03.938976+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:03.938976+0000","last_peered":"2024-04-26T02:33:03.938976+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:03.938976+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9826,"log_dups_size":0,"ondisk_log_size":9826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16429580,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9824,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9580","reported_seq":9750,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.064958+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:01.064958+0000","last_peered":"2024-04-26T02:33:01.064958+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:01.064958+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16791838,"num_objects":2051,"num_object_clones":0,"num_object_copies":8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9579,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":67}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9420","reported_seq":13446,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:00.803235+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:00.803235+0000","last_peered":"2024-04-26T02:33:00.803235+0000","last_clean":"2024-04-26T02:33:00.803235+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:00.803235+0000","last_undegraded":"2024-04-26T02:33:00.803235+0000","last_fullsized":"2024-04-26T02:33:00.803235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9420,"log_dups_size":0,"ondisk_log_size":9420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16871948,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9529","reported_seq":9760,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.060747+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:02.060747+0000","last_peered":"2024-04-26T02:33:02.060747+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:02.060747+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16937484,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9667","reported_seq":9923,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.038346+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:02.038346+0000","last_peered":"2024-04-26T02:33:02.038346+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:02.038346+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15835136,"num_objects":1933,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9762","reported_seq":14015,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:04.227477+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:04.227477+0000","last_peered":"2024-04-26T02:33:04.227477+0000","last_clean":"2024-04-26T02:33:04.227477+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:04.227477+0000","last_undegraded":"2024-04-26T02:33:04.227477+0000","last_fullsized":"2024-04-26T02:33:04.227477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:07.997510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16347660,"num_objects":1996,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9760,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10010","reported_seq":10265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:04.171990+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:04.171990+0000","last_peered":"2024-04-26T02:33:04.171990+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:04.171990+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208396,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9679","reported_seq":13184,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.372670+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:02.372670+0000","last_peered":"2024-04-26T02:33:02.372670+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:02.372670+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16876568,"num_objects":2061,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1721,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9614","reported_seq":9788,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.057160+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:01.057160+0000","last_peered":"2024-04-26T02:33:01.057160+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:01.057160+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15970828,"num_objects":1950,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-45}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9694","reported_seq":9929,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.080834+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:02.080834+0000","last_peered":"2024-04-26T02:33:02.080834+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:02.080834+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16583442,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9602","reported_seq":9832,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.010021+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:02.010021+0000","last_peered":"2024-04-26T02:33:02.010021+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:02.010021+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16341254,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":97}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9738","reported_seq":9956,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:01.090260+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:01.090260+0000","last_peered":"2024-04-26T02:33:01.090260+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:01.090260+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9738,"log_dups_size":0,"ondisk_log_size":9738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16718086,"num_objects":2041,"num_object_clones":0,"num_object_copies":8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9737,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-4}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10044","reported_seq":10291,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:02.056402+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:02.056402+0000","last_peered":"2024-04-26T02:33:02.056402+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:02.056402+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17019404,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9793","reported_seq":14003,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:00.519838+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:00.519838+0000","last_peered":"2024-04-26T02:33:00.519838+0000","last_clean":"2024-04-26T02:33:00.519838+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:00.519838+0000","last_undegraded":"2024-04-26T02:33:00.519838+0000","last_fullsized":"2024-04-26T02:33:00.519838+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:03.609797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16888332,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9818","reported_seq":10047,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:04.276163+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:04.276163+0000","last_peered":"2024-04-26T02:33:04.276163+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:04.276163+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208396,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-8}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":264789184,"num_objects":32330,"num_object_clones":0,"num_object_copies":129320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155227,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8782,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":530386944,"data_stored":530386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155254,"ondisk_log_size":155254,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117101,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680624,"kb_used_data":129704,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93691216,"statfs":{"total":96636764160,"available":95939805184,"internally_reserved":0,"allocated":132816896,"data_stored":132613234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":564129488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,2,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703736,"kb_used_data":131108,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93668104,"statfs":{"total":96636764160,"available":95916138496,"internally_reserved":0,"allocated":134254592,"data_stored":134050930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586160,"kb_used_data":131112,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93785680,"statfs":{"total":96636764160,"available":96036536320,"internally_reserved":0,"allocated":134258688,"data_stored":134055026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":465956559},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,4,4,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723692,"kb_used_data":128420,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93648148,"statfs":{"total":96636764160,"available":95895703552,"internally_reserved":0,"allocated":131502080,"data_stored":131298418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130912256,"data_stored":130912256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133763072,"data_stored":133763072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133287936,"data_stored":133287936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":132423680,"data_stored":132423680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9478", 'reported_seq': 9710, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:01.887389+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:01.887389+0000', 'last_peered': '2024-04-26T02:33:01.887389+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:01.887389+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9478, 'log_dups_size': 0, 'ondisk_log_size': 9478, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16760832, 'num_objects': 2046, 'num_object_clones': 0, 'num_object_copies': 8184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9478, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9826", 'reported_seq': 10054, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:03.938976+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:33:03.938976+0000', 'last_peered': '2024-04-26T02:33:03.938976+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:03.938976+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9826, 'log_dups_size': 0, 'ondisk_log_size': 9826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16429580, 'num_objects': 2006, 'num_object_clones': 0, 'num_object_copies': 8024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9824, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9580", 'reported_seq': 9750, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:01.064958+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:01.064958+0000', 'last_peered': '2024-04-26T02:33:01.064958+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:01.064958+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9580, 'log_dups_size': 0, 'ondisk_log_size': 9580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16791838, 'num_objects': 2051, 'num_object_clones': 0, 'num_object_copies': 8204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2051, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9579, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 67}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9529", 'reported_seq': 9760, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.060747+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:02.060747+0000', 'last_peered': '2024-04-26T02:33:02.060747+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:02.060747+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9529, 'log_dups_size': 0, 'ondisk_log_size': 9529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16937484, 'num_objects': 2068, 'num_object_clones': 0, 'num_object_copies': 8272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9527, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:06.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9667", 'reported_seq': 9923, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.038346+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:02.038346+0000', 'last_peered': '2024-04-26T02:33:02.038346+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:02.038346+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9667, 'log_dups_size': 0, 'ondisk_log_size': 9667, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15835136, 'num_objects': 1933, 'num_object_clones': 0, 'num_object_copies': 7732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9667, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -17}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'10010", 'reported_seq': 10265, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:04.171990+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:04.171990+0000', 'last_peered': '2024-04-26T02:33:04.171990+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:04.171990+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16208396, 'num_objects': 1979, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10008, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -73}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9679", 'reported_seq': 13184, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.372670+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:33:02.372670+0000', 'last_peered': '2024-04-26T02:33:02.372670+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:33:02.372670+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9679, 'log_dups_size': 0, 'ondisk_log_size': 9679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16876568, 'num_objects': 2061, 'num_object_clones': 0, 'num_object_copies': 8244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9676, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1721, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1772}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9614", 'reported_seq': 9788, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:01.057160+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:01.057160+0000', 'last_peered': '2024-04-26T02:33:01.057160+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:01.057160+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9614, 'log_dups_size': 0, 'ondisk_log_size': 9614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15970828, 'num_objects': 1950, 'num_object_clones': 0, 'num_object_copies': 7800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9612, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -45}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9694", 'reported_seq': 9929, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.080834+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:02.080834+0000', 'last_peered': '2024-04-26T02:33:02.080834+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:02.080834+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9694, 'log_dups_size': 0, 'ondisk_log_size': 9694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16583442, 'num_objects': 2025, 'num_object_clones': 0, 'num_object_copies': 8100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9691, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 91}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:06.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9602", 'reported_seq': 9832, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.010021+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:02.010021+0000', 'last_peered': '2024-04-26T02:33:02.010021+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:02.010021+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9602, 'log_dups_size': 0, 'ondisk_log_size': 9602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16341254, 'num_objects': 1995, 'num_object_clones': 0, 'num_object_copies': 7980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9601, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 97}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9738", 'reported_seq': 9956, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:01.090260+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:01.090260+0000', 'last_peered': '2024-04-26T02:33:01.090260+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:01.090260+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9738, 'log_dups_size': 0, 'ondisk_log_size': 9738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16718086, 'num_objects': 2041, 'num_object_clones': 0, 'num_object_copies': 8164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9737, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -4}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'10044", 'reported_seq': 10291, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:02.056402+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:02.056402+0000', 'last_peered': '2024-04-26T02:33:02.056402+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:02.056402+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17019404, 'num_objects': 2078, 'num_object_clones': 0, 'num_object_copies': 8312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10042, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 99}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:06.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9818", 'reported_seq': 10047, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:04.276163+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:04.276163+0000', 'last_peered': '2024-04-26T02:33:04.276163+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:04.276163+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9818, 'log_dups_size': 0, 'ondisk_log_size': 9818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16208396, 'num_objects': 1979, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9816, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -8}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:06.384 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:08.258 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:08.259 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23987/128064 objects degraded (18.730%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":584,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32016,"data_bytes":262216896,"bytes_used":2765307904,"bytes_avail":383781748736,"bytes_total":386547056640,"degraded_objects":23987,"degraded_total":128064,"degraded_ratio":0.18730478510744628,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":187},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:33:08.413 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:08.414 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:10.104 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:10.105 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:10.287 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-26T02:33:08.692514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258882752,"num_objects":31609,"num_object_clones":0,"num_object_copies":126436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155948,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8866,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155875,"ondisk_log_size":155875,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737048,"kb_used_data":509008,"kb_used_omap":14,"kb_used_meta":2227953,"kb_avail":374750312,"statfs":{"total":386547056640,"available":383744319488,"internally_reserved":0,"allocated":521224192,"data_stored":520409544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2281424435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,9,6,3,5,10,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19234816,"num_objects":-2348,"num_object_clones":0,"num_object_copies":-9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2348,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005493"},"pg_stats":[{"pgid":"1.f","version":"38'9539","reported_seq":9771,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.356700+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:07.356700+0000","last_peered":"2024-04-26T02:33:07.356700+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:07.356700+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9539,"log_dups_size":0,"ondisk_log_size":9539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16261120,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":58}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9826","reported_seq":10054,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:03.938976+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:03.938976+0000","last_peered":"2024-04-26T02:33:03.938976+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:03.938976+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9826,"log_dups_size":0,"ondisk_log_size":9826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16429580,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9824,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9643","reported_seq":9813,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:05.922566+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:05.922566+0000","last_peered":"2024-04-26T02:33:05.922566+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:05.922566+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9643,"log_dups_size":0,"ondisk_log_size":9643,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16275742,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9642,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9482","reported_seq":13507,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:05.899869+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:05.899869+0000","last_peered":"2024-04-26T02:33:05.899869+0000","last_clean":"2024-04-26T02:33:05.899869+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:05.899869+0000","last_undegraded":"2024-04-26T02:33:05.899869+0000","last_fullsized":"2024-04-26T02:33:05.899869+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9482,"log_dups_size":0,"ondisk_log_size":9482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:06.619800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16364044,"num_objects":1998,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9480,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9586","reported_seq":9817,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.296525+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:07.296525+0000","last_peered":"2024-04-26T02:33:07.296525+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:07.296525+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9586,"log_dups_size":0,"ondisk_log_size":9586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16470540,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9584,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9712","reported_seq":9968,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.234208+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:07.234208+0000","last_peered":"2024-04-26T02:33:07.234208+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:07.234208+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-62}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9762","reported_seq":14015,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:04.227477+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:04.227477+0000","last_peered":"2024-04-26T02:33:04.227477+0000","last_clean":"2024-04-26T02:33:04.227477+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:04.227477+0000","last_undegraded":"2024-04-26T02:33:04.227477+0000","last_fullsized":"2024-04-26T02:33:04.227477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:07.997510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16347660,"num_objects":1996,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9760,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10010","reported_seq":10265,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:04.171990+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:04.171990+0000","last_peered":"2024-04-26T02:33:04.171990+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:04.171990+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208396,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9743","reported_seq":13416,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.472784+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:07.472784+0000","last_peered":"2024-04-26T02:33:07.472784+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:07.472784+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9743,"log_dups_size":0,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16352280,"num_objects":1997,"num_object_clones":0,"num_object_copies":7988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1805,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9680","reported_seq":9854,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:06.035595+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:06.035595+0000","last_peered":"2024-04-26T02:33:06.035595+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:06.035595+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15430156,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9759","reported_seq":9994,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.425529+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:07.425529+0000","last_peered":"2024-04-26T02:33:07.425529+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:07.425529+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16050962,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9658","reported_seq":9888,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.359991+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:07.359991+0000","last_peered":"2024-04-26T02:33:07.359991+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:07.359991+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9658,"log_dups_size":0,"ondisk_log_size":9658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15882502,"num_objects":1939,"num_object_clones":0,"num_object_copies":7756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":41}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9792","reported_seq":10010,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:06.135327+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:06.135327+0000","last_peered":"2024-04-26T02:33:06.135327+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:06.135327+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16275718,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10103","reported_seq":10351,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:07.476645+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:07.476645+0000","last_peered":"2024-04-26T02:33:07.476645+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:07.476645+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16536076,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":40}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9862","reported_seq":14070,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:05.954509+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:05.954509+0000","last_peered":"2024-04-26T02:33:05.954509+0000","last_clean":"2024-04-26T02:33:05.954509+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:05.954509+0000","last_undegraded":"2024-04-26T02:33:05.954509+0000","last_fullsized":"2024-04-26T02:33:05.954509+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9862,"log_dups_size":0,"ondisk_log_size":9862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:08.686563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16323084,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9860,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9818","reported_seq":10047,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:04.276163+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:04.276163+0000","last_peered":"2024-04-26T02:33:04.276163+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:04.276163+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208396,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-8}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258882752,"num_objects":31609,"num_object_clones":0,"num_object_copies":126436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155948,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8866,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":518479872,"data_stored":518479872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155875,"ondisk_log_size":155875,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717176,"kb_used_data":125972,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93654664,"statfs":{"total":96636764160,"available":95902375936,"internally_reserved":0,"allocated":128995328,"data_stored":128791666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726320,"kb_used_data":127332,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93645520,"statfs":{"total":96636764160,"available":95893012480,"internally_reserved":0,"allocated":130387968,"data_stored":130184306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,0,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569860,"kb_used_data":127284,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93801980,"statfs":{"total":96636764160,"available":96053227520,"internally_reserved":0,"allocated":130338816,"data_stored":130135154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":453177039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,0,2,2,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723692,"kb_used_data":128420,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93648148,"statfs":{"total":96636764160,"available":95895703552,"internally_reserved":0,"allocated":131502080,"data_stored":131298418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130912256,"data_stored":130912256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129810432,"data_stored":129810432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129425408,"data_stored":129425408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128331776,"data_stored":128331776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:13.289 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:14.680 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:14.681 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:14.814 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-26T02:33:12.699034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250633408,"num_objects":30602,"num_object_clones":0,"num_object_copies":122408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156955,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8926,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156882,"ondisk_log_size":156882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2747636,"kb_used_data":493260,"kb_used_omap":14,"kb_used_meta":2254321,"kb_avail":374739724,"statfs":{"total":386547056640,"available":383733477376,"internally_reserved":0,"allocated":505098240,"data_stored":504283592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2308425267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,5,8,18,14,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21921792,"num_objects":-2676,"num_object_clones":0,"num_object_copies":-10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008518"},"pg_stats":[{"pgid":"1.f","version":"38'9592","reported_seq":9824,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.394640+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:12.394640+0000","last_peered":"2024-04-26T02:33:12.394640+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:12.394640+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15826944,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9881","reported_seq":10109,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:09.149995+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:09.149995+0000","last_peered":"2024-04-26T02:33:09.149995+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:09.149995+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15979020,"num_objects":1951,"num_object_clones":0,"num_object_copies":7804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9879,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-49}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9702","reported_seq":9872,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:11.083443+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:11.083443+0000","last_peered":"2024-04-26T02:33:11.083443+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:11.083443+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15792414,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-55}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9551","reported_seq":13575,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:10.993627+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:10.993627+0000","last_peered":"2024-04-26T02:33:10.993627+0000","last_clean":"2024-04-26T02:33:10.993627+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:10.993627+0000","last_undegraded":"2024-04-26T02:33:10.993627+0000","last_fullsized":"2024-04-26T02:33:10.993627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:11.746554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15798796,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9651","reported_seq":9879,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.298767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:12.298767+0000","last_peered":"2024-04-26T02:33:12.298767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:12.298767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15938060,"num_objects":1946,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9771","reported_seq":10028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.397164+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:12.397164+0000","last_peered":"2024-04-26T02:33:12.397164+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:12.397164+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14983168,"num_objects":1829,"num_object_clones":0,"num_object_copies":7316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9841","reported_seq":14094,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:09.226677+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:09.226677+0000","last_peered":"2024-04-26T02:33:09.226677+0000","last_clean":"2024-04-26T02:33:09.226677+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:09.226677+0000","last_undegraded":"2024-04-26T02:33:09.226677+0000","last_fullsized":"2024-04-26T02:33:09.226677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:13.914153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15700492,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9839,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10071","reported_seq":10326,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:09.134364+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:09.134364+0000","last_peered":"2024-04-26T02:33:09.134364+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:09.134364+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15708684,"num_objects":1918,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9813","reported_seq":13606,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.513123+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:12.513123+0000","last_peered":"2024-04-26T02:33:12.513123+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:12.513123+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15778840,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9734","reported_seq":9908,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:10.889911+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:10.889911+0000","last_peered":"2024-04-26T02:33:10.889911+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:10.889911+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9734,"log_dups_size":0,"ondisk_log_size":9734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14987788,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9839","reported_seq":10074,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.444242+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:12.444242+0000","last_peered":"2024-04-26T02:33:12.444242+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:12.444242+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9839,"log_dups_size":0,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-54}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9722","reported_seq":9952,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.385573+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:12.385573+0000","last_peered":"2024-04-26T02:33:12.385573+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:12.385573+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15358214,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-23}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9859","reported_seq":10077,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:11.195606+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:11.195606+0000","last_peered":"2024-04-26T02:33:11.195606+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:11.195606+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15726854,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9858,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10161","reported_seq":10408,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.399324+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:12.399324+0000","last_peered":"2024-04-26T02:33:12.399324+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:12.399324+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16060940,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-18}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9918","reported_seq":14127,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:10.961514+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:10.961514+0000","last_peered":"2024-04-26T02:33:10.961514+0000","last_clean":"2024-04-26T02:33:10.961514+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:10.961514+0000","last_undegraded":"2024-04-26T02:33:10.961514+0000","last_fullsized":"2024-04-26T02:33:10.961514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:13.720754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15864332,"num_objects":1937,"num_object_clones":0,"num_object_copies":7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9876","reported_seq":10106,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:09.228979+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:09.228979+0000","last_peered":"2024-04-26T02:33:09.228979+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:09.228979+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15733260,"num_objects":1921,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9874,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-66}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":250633408,"num_objects":30602,"num_object_clones":0,"num_object_copies":122408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156955,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8926,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":502026240,"data_stored":502026240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156882,"ondisk_log_size":156882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721532,"kb_used_data":122028,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93650308,"statfs":{"total":96636764160,"available":95897915392,"internally_reserved":0,"allocated":124956672,"data_stored":124753010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709824,"kb_used_data":123320,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93662016,"statfs":{"total":96636764160,"available":95909904384,"internally_reserved":0,"allocated":126279680,"data_stored":126076018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592312,"kb_used_data":123376,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93779528,"statfs":{"total":96636764160,"available":96030236672,"internally_reserved":0,"allocated":126337024,"data_stored":126133362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723968,"kb_used_data":124536,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93647872,"statfs":{"total":96636764160,"available":95895420928,"internally_reserved":0,"allocated":127524864,"data_stored":127321202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,4,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":126865408,"data_stored":126865408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125673472,"data_stored":125673472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125304832,"data_stored":125304832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124182528,"data_stored":124182528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:14.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:16.532 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:16.627 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-26T02:33:14.699568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248601792,"num_objects":30354,"num_object_clones":0,"num_object_copies":121416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30354,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157203,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8926,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157030,"ondisk_log_size":157030,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735320,"kb_used_data":489264,"kb_used_omap":14,"kb_used_meta":2246001,"kb_avail":374752040,"statfs":{"total":386547056640,"available":383746088960,"internally_reserved":0,"allocated":501006336,"data_stored":500191688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2299905587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,10,14,13,16,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17940480,"num_objects":-2190,"num_object_clones":0,"num_object_copies":-8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008568"},"pg_stats":[{"pgid":"1.f","version":"38'9592","reported_seq":9824,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.394640+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:12.394640+0000","last_peered":"2024-04-26T02:33:12.394640+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:12.394640+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15826944,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9946","reported_seq":10174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:14.190519+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:14.190519+0000","last_peered":"2024-04-26T02:33:14.190519+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:14.190519+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446540,"num_objects":1886,"num_object_clones":0,"num_object_copies":7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9702","reported_seq":9872,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:11.083443+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:11.083443+0000","last_peered":"2024-04-26T02:33:11.083443+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:11.083443+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15792414,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9701,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-55}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9551","reported_seq":13575,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:10.993627+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:10.993627+0000","last_peered":"2024-04-26T02:33:10.993627+0000","last_clean":"2024-04-26T02:33:10.993627+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:10.993627+0000","last_undegraded":"2024-04-26T02:33:10.993627+0000","last_fullsized":"2024-04-26T02:33:10.993627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:11.746554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15798796,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9651","reported_seq":9879,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.298767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:12.298767+0000","last_peered":"2024-04-26T02:33:12.298767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:12.298767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15938060,"num_objects":1946,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9771","reported_seq":10028,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.397164+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:12.397164+0000","last_peered":"2024-04-26T02:33:12.397164+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:12.397164+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14983168,"num_objects":1829,"num_object_clones":0,"num_object_copies":7316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9898","reported_seq":14149,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:14.073117+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:14.073117+0000","last_peered":"2024-04-26T02:33:14.073117+0000","last_clean":"2024-04-26T02:33:14.073117+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:14.073117+0000","last_undegraded":"2024-04-26T02:33:14.073117+0000","last_fullsized":"2024-04-26T02:33:14.073117+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9898,"log_dups_size":0,"ondisk_log_size":9898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15233548,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10141","reported_seq":10396,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:14.176101+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:14.176101+0000","last_peered":"2024-04-26T02:33:14.176101+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:14.176101+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135244,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10139,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9813","reported_seq":13606,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.513123+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:12.513123+0000","last_peered":"2024-04-26T02:33:12.513123+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:12.513123+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15778840,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9734","reported_seq":9908,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:10.889911+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:10.889911+0000","last_peered":"2024-04-26T02:33:10.889911+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:10.889911+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9734,"log_dups_size":0,"ondisk_log_size":9734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14987788,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9839","reported_seq":10074,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.444242+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:12.444242+0000","last_peered":"2024-04-26T02:33:12.444242+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:12.444242+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9839,"log_dups_size":0,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15395602,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-54}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9722","reported_seq":9952,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.385573+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:12.385573+0000","last_peered":"2024-04-26T02:33:12.385573+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:12.385573+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15358214,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-23}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9859","reported_seq":10077,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:11.195606+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:11.195606+0000","last_peered":"2024-04-26T02:33:11.195606+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:11.195606+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15726854,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9858,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10161","reported_seq":10408,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:12.399324+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:12.399324+0000","last_peered":"2024-04-26T02:33:12.399324+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:12.399324+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16060940,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-18}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9918","reported_seq":14127,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:10.961514+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:10.961514+0000","last_peered":"2024-04-26T02:33:10.961514+0000","last_clean":"2024-04-26T02:33:10.961514+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:10.961514+0000","last_undegraded":"2024-04-26T02:33:10.961514+0000","last_fullsized":"2024-04-26T02:33:10.961514+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:13.720754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15864332,"num_objects":1937,"num_object_clones":0,"num_object_copies":7748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9932","reported_seq":10162,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:13.763308+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:13.763308+0000","last_peered":"2024-04-26T02:33:13.763308+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:13.763308+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15274508,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":248601792,"num_objects":30354,"num_object_clones":0,"num_object_copies":121416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30354,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157203,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8926,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":497876992,"data_stored":497876992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157030,"ondisk_log_size":157030,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721532,"kb_used_data":122028,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93650308,"statfs":{"total":96636764160,"available":95897915392,"internally_reserved":0,"allocated":124956672,"data_stored":124753010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709824,"kb_used_data":123320,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93662016,"statfs":{"total":96636764160,"available":95909904384,"internally_reserved":0,"allocated":126279680,"data_stored":126076018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592312,"kb_used_data":123376,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93779528,"statfs":{"total":96636764160,"available":96030236672,"internally_reserved":0,"allocated":126337024,"data_stored":126133362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711652,"kb_used_data":120540,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93660188,"statfs":{"total":96636764160,"available":95908032512,"internally_reserved":0,"allocated":123432960,"data_stored":123229298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":122716160,"data_stored":122716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125673472,"data_stored":125673472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125304832,"data_stored":125304832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124182528,"data_stored":124182528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9592", 'reported_seq': 9824, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.394640+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:12.394640+0000', 'last_peered': '2024-04-26T02:33:12.394640+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:12.394640+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9592, 'log_dups_size': 0, 'ondisk_log_size': 9592, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15826944, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9592, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "38'9946", 'reported_seq': 10174, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:14.190519+0000', 'last_change': '2024-04-26T02:24:59.567470+0000', 'last_active': '2024-04-26T02:33:14.190519+0000', 'last_peered': '2024-04-26T02:33:14.190519+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:14.190519+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9946, 'log_dups_size': 0, 'ondisk_log_size': 9946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15446540, 'num_objects': 1886, 'num_object_clones': 0, 'num_object_copies': 7544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2002, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9944, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "38'9702", 'reported_seq': 9872, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:11.083443+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:11.083443+0000', 'last_peered': '2024-04-26T02:33:11.083443+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:11.083443+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9702, 'log_dups_size': 0, 'ondisk_log_size': 9702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15792414, 'num_objects': 1929, 'num_object_clones': 0, 'num_object_copies': 7716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1929, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9701, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -55}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:16.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9651", 'reported_seq': 9879, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.298767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:12.298767+0000', 'last_peered': '2024-04-26T02:33:12.298767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:12.298767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9651, 'log_dups_size': 0, 'ondisk_log_size': 9651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15938060, 'num_objects': 1946, 'num_object_clones': 0, 'num_object_copies': 7784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9649, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 17}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9771", 'reported_seq': 10028, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.397164+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:12.397164+0000', 'last_peered': '2024-04-26T02:33:12.397164+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:12.397164+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9771, 'log_dups_size': 0, 'ondisk_log_size': 9771, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14983168, 'num_objects': 1829, 'num_object_clones': 0, 'num_object_copies': 7316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9771, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -121}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "38'10141", 'reported_seq': 10396, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:14.176101+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:14.176101+0000', 'last_peered': '2024-04-26T02:33:14.176101+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:14.176101+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15135244, 'num_objects': 1848, 'num_object_clones': 0, 'num_object_copies': 7392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10139, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9813", 'reported_seq': 13606, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.513123+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:33:12.513123+0000', 'last_peered': '2024-04-26T02:33:12.513123+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:33:12.513123+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15778840, 'num_objects': 1927, 'num_object_clones': 0, 'num_object_copies': 7708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 145, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9810, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1865, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "38'9734", 'reported_seq': 9908, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:10.889911+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:10.889911+0000', 'last_peered': '2024-04-26T02:33:10.889911+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:10.889911+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9734, 'log_dups_size': 0, 'ondisk_log_size': 9734, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14987788, 'num_objects': 1830, 'num_object_clones': 0, 'num_object_copies': 7320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9732, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -165}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:16.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9839", 'reported_seq': 10074, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.444242+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:12.444242+0000', 'last_peered': '2024-04-26T02:33:12.444242+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:12.444242+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9839, 'log_dups_size': 0, 'ondisk_log_size': 9839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15395602, 'num_objects': 1880, 'num_object_clones': 0, 'num_object_copies': 7520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9836, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -54}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9722", 'reported_seq': 9952, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.385573+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:12.385573+0000', 'last_peered': '2024-04-26T02:33:12.385573+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:12.385573+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9722, 'log_dups_size': 0, 'ondisk_log_size': 9722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15358214, 'num_objects': 1875, 'num_object_clones': 0, 'num_object_copies': 7500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9721, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -23}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "38'9859", 'reported_seq': 10077, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:11.195606+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:11.195606+0000', 'last_peered': '2024-04-26T02:33:11.195606+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:11.195606+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9859, 'log_dups_size': 0, 'ondisk_log_size': 9859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15726854, 'num_objects': 1920, 'num_object_clones': 0, 'num_object_copies': 7680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9858, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'10161", 'reported_seq': 10408, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:12.399324+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:12.399324+0000', 'last_peered': '2024-04-26T02:33:12.399324+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:12.399324+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16060940, 'num_objects': 1961, 'num_object_clones': 0, 'num_object_copies': 7844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10159, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -18}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:16.630 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "38'9932", 'reported_seq': 10162, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:13.763308+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:13.763308+0000', 'last_peered': '2024-04-26T02:33:13.763308+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:13.763308+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9932, 'log_dups_size': 0, 'ondisk_log_size': 9932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15274508, 'num_objects': 1865, 'num_object_clones': 0, 'num_object_copies': 7460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9930, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:16.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:18.528 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:18.528 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23843/120116 objects degraded (19.850%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":595,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30029,"data_bytes":245939392,"bytes_used":2801238016,"bytes_avail":383745818624,"bytes_total":386547056640,"degraded_objects":23843,"degraded_total":120116,"degraded_ratio":0.19849978354257553,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:33:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:18.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:20.102 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:20.103 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:20.117 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-04-26T02:33:18.700692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242277568,"num_objects":29582,"num_object_clones":0,"num_object_copies":118328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157975,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8968,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157702,"ondisk_log_size":157702,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735820,"kb_used_data":477264,"kb_used_omap":14,"kb_used_meta":2258481,"kb_avail":374751540,"statfs":{"total":386547056640,"available":383745576960,"internally_reserved":0,"allocated":488718336,"data_stored":487903688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2312685107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,6,12,11,10,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19939328,"num_objects":-2434,"num_object_clones":0,"num_object_copies":-9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008702"},"pg_stats":[{"pgid":"1.f","version":"38'9661","reported_seq":9893,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.449970+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:17.449970+0000","last_peered":"2024-04-26T02:33:17.449970+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:17.449970+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15261696,"num_objects":1863,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'9946","reported_seq":10174,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:14.190519+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:14.190519+0000","last_peered":"2024-04-26T02:33:14.190519+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:14.190519+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446540,"num_objects":1886,"num_object_clones":0,"num_object_copies":7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9756","reported_seq":9923,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:16.173037+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:16.173037+0000","last_peered":"2024-04-26T02:33:16.173037+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:16.173037+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15350046,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9755,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9619","reported_seq":13644,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:16.143795+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:16.143795+0000","last_peered":"2024-04-26T02:33:16.143795+0000","last_clean":"2024-04-26T02:33:16.143795+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:16.143795+0000","last_undegraded":"2024-04-26T02:33:16.143795+0000","last_fullsized":"2024-04-26T02:33:16.143795+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9619,"log_dups_size":0,"ondisk_log_size":9619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:17.655329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15241740,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9617,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9704","reported_seq":9935,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.455984+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:17.455984+0000","last_peered":"2024-04-26T02:33:17.455984+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:17.455984+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15503884,"num_objects":1893,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9846","reported_seq":10100,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.507200+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:17.507200+0000","last_peered":"2024-04-26T02:33:17.507200+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:17.507200+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14368768,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9898","reported_seq":14149,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:14.073117+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:14.073117+0000","last_peered":"2024-04-26T02:33:14.073117+0000","last_clean":"2024-04-26T02:33:14.073117+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:14.073117+0000","last_undegraded":"2024-04-26T02:33:14.073117+0000","last_fullsized":"2024-04-26T02:33:14.073117+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9898,"log_dups_size":0,"ondisk_log_size":9898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15233548,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10141","reported_seq":10396,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:14.176101+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:14.176101+0000","last_peered":"2024-04-26T02:33:14.176101+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:14.176101+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15135244,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10139,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9889","reported_seq":13763,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.536710+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:17.536710+0000","last_peered":"2024-04-26T02:33:17.536710+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:17.536710+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15156248,"num_objects":1851,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9886,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1748}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9796","reported_seq":9970,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:16.042298+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:16.042298+0000","last_peered":"2024-04-26T02:33:16.042298+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:16.042298+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14479884,"num_objects":1768,"num_object_clones":0,"num_object_copies":7072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9891","reported_seq":10126,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.326055+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:17.326055+0000","last_peered":"2024-04-26T02:33:17.326055+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:17.326055+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14969618,"num_objects":1828,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9790","reported_seq":10019,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.451145+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:17.451145+0000","last_peered":"2024-04-26T02:33:17.451145+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:17.451145+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14801158,"num_objects":1807,"num_object_clones":0,"num_object_copies":7228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-91}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9929","reported_seq":10147,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:16.210752+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:16.210752+0000","last_peered":"2024-04-26T02:33:16.210752+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:16.210752+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9929,"log_dups_size":0,"ondisk_log_size":9929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15153414,"num_objects":1850,"num_object_clones":0,"num_object_copies":7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9928,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-195}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10215","reported_seq":10463,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:17.471210+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:17.471210+0000","last_peered":"2024-04-26T02:33:17.471210+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:17.471210+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15618572,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-72}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'9989","reported_seq":14199,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:16.231986+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:16.231986+0000","last_peered":"2024-04-26T02:33:16.231986+0000","last_clean":"2024-04-26T02:33:16.231986+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:16.231986+0000","last_undegraded":"2024-04-26T02:33:16.231986+0000","last_fullsized":"2024-04-26T02:33:16.231986+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:19.682578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15282700,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9932","reported_seq":10162,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:13.763308+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:13.763308+0000","last_peered":"2024-04-26T02:33:13.763308+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:13.763308+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15274508,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242277568,"num_objects":29582,"num_object_clones":0,"num_object_copies":118328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157975,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8968,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":485543936,"data_stored":485543936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157702,"ondisk_log_size":157702,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721768,"kb_used_data":118112,"kb_used_omap":4,"kb_used_meta":603643,"kb_avail":93650072,"statfs":{"total":96636764160,"available":95897673728,"internally_reserved":0,"allocated":120946688,"data_stored":120743026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":618131152},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,1,9,2,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709944,"kb_used_data":119276,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93661896,"statfs":{"total":96636764160,"available":95909781504,"internally_reserved":0,"allocated":122138624,"data_stored":121934962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3,7,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592456,"kb_used_data":119336,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93779384,"statfs":{"total":96636764160,"available":96030089216,"internally_reserved":0,"allocated":122200064,"data_stored":121996402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711652,"kb_used_data":120540,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93660188,"statfs":{"total":96636764160,"available":95908032512,"internally_reserved":0,"allocated":123432960,"data_stored":123229298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":122716160,"data_stored":122716160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121458688,"data_stored":121458688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121192448,"data_stored":121192448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120176640,"data_stored":120176640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:23.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:24.640 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:24.640 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:24.737 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-04-26T02:33:24.329541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234388672,"num_objects":28619,"num_object_clones":0,"num_object_copies":114476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158938,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158565,"ondisk_log_size":158565,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703860,"kb_used_data":461972,"kb_used_omap":14,"kb_used_meta":2241841,"kb_avail":374783500,"statfs":{"total":386547056640,"available":383778304000,"internally_reserved":0,"allocated":473059328,"data_stored":472244680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2295645747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,13,13,11,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14213120,"num_objects":-1735,"num_object_clones":0,"num_object_copies":-6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1735,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.629972"},"pg_stats":[{"pgid":"1.f","version":"38'9722","reported_seq":9951,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.570911+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:22.570911+0000","last_peered":"2024-04-26T02:33:22.570911+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:22.570911+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14761984,"num_objects":1802,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'10008","reported_seq":10234,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:19.394117+0000","last_change":"2024-04-26T02:24:59.567470+0000","last_active":"2024-04-26T02:33:19.394117+0000","last_peered":"2024-04-26T02:33:19.394117+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:19.394117+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14938636,"num_objects":1824,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10006,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"38'9813","reported_seq":9983,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:21.445725+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:21.445725+0000","last_peered":"2024-04-26T02:33:21.445725+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:21.445725+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14883102,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9812,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"38'9663","reported_seq":13689,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:20.835523+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:20.835523+0000","last_peered":"2024-04-26T02:33:20.835523+0000","last_clean":"2024-04-26T02:33:20.835523+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:20.835523+0000","last_undegraded":"2024-04-26T02:33:20.835523+0000","last_fullsized":"2024-04-26T02:33:20.835523+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:22.732244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14881292,"num_objects":1817,"num_object_clones":0,"num_object_copies":7268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9766","reported_seq":9997,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.555296+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:22.555296+0000","last_peered":"2024-04-26T02:33:22.555296+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:22.555296+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14995980,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-98}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9902","reported_seq":10159,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.278954+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:22.278954+0000","last_peered":"2024-04-26T02:33:22.278954+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:22.278954+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13910016,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"38'9969","reported_seq":14222,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:19.375154+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:19.375154+0000","last_peered":"2024-04-26T02:33:19.375154+0000","last_clean":"2024-04-26T02:33:19.375154+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:19.375154+0000","last_undegraded":"2024-04-26T02:33:19.375154+0000","last_fullsized":"2024-04-26T02:33:19.375154+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:19.897945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14651916,"num_objects":1789,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'10211","reported_seq":10466,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:19.292017+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:19.292017+0000","last_peered":"2024-04-26T02:33:19.292017+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:19.292017+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14561804,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10209,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-274}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9952","reported_seq":14023,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.491037+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:22.491037+0000","last_peered":"2024-04-26T02:33:22.491037+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:22.491037+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14640152,"num_objects":1788,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2004,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'9857","reported_seq":10031,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:21.217517+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:21.217517+0000","last_peered":"2024-04-26T02:33:21.217517+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:21.217517+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9857,"log_dups_size":0,"ondisk_log_size":9857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980172,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-288}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9948","reported_seq":10183,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.404532+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:22.404532+0000","last_peered":"2024-04-26T02:33:22.404532+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:22.404532+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14502674,"num_objects":1771,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9849","reported_seq":10077,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.491069+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:22.491069+0000","last_peered":"2024-04-26T02:33:22.491069+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:22.491069+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14317830,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'9987","reported_seq":10205,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:21.460850+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:21.460850+0000","last_peered":"2024-04-26T02:33:21.460850+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:21.460850+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14678278,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9986,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-253}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10276","reported_seq":10524,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.406626+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:22.406626+0000","last_peered":"2024-04-26T02:33:22.406626+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:22.406626+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15118860,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-133}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"38'10050","reported_seq":14261,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-26T02:33:21.183216+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:21.183216+0000","last_peered":"2024-04-26T02:33:21.183216+0000","last_clean":"2024-04-26T02:33:21.183216+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:21.183216+0000","last_undegraded":"2024-04-26T02:33:21.183216+0000","last_fullsized":"2024-04-26T02:33:21.183216+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:25.642037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14782988,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"38'9992","reported_seq":10222,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:19.400442+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:19.400442+0000","last_peered":"2024-04-26T02:33:19.400442+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:19.400442+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9992,"log_dups_size":0,"ondisk_log_size":9992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14782988,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9990,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":234388672,"num_objects":28619,"num_object_clones":0,"num_object_copies":114476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158938,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9065,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":469381120,"data_stored":469381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158565,"ondisk_log_size":158565,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705564,"kb_used_data":114388,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93666276,"statfs":{"total":96636764160,"available":95914266624,"internally_reserved":0,"allocated":117133312,"data_stored":116929650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,6,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710240,"kb_used_data":115416,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93661600,"statfs":{"total":96636764160,"available":95909478400,"internally_reserved":0,"allocated":118185984,"data_stored":117982322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3,7,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576196,"kb_used_data":115580,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93795644,"statfs":{"total":96636764160,"available":96046739456,"internally_reserved":0,"allocated":118353920,"data_stored":118150258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711860,"kb_used_data":116588,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93659980,"statfs":{"total":96636764160,"available":95907819520,"internally_reserved":0,"allocated":119386112,"data_stored":119182450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118636544,"data_stored":118636544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117456896,"data_stored":117456896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117096448,"data_stored":117096448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":116191232,"data_stored":116191232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:24.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:26.742 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:26.743 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:26.901 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-04-26T02:33:26.704257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230055104,"num_objects":28090,"num_object_clones":0,"num_object_copies":112360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28090,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159467,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9089,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158994,"ondisk_log_size":158994,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704444,"kb_used_data":450072,"kb_used_omap":14,"kb_used_meta":2254321,"kb_avail":374782916,"statfs":{"total":386547056640,"available":383777705984,"internally_reserved":0,"allocated":460873728,"data_stored":460035638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2308425267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,11,10,6,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12222464,"num_objects":-1492,"num_object_clones":0,"num_object_copies":-5968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003564"},"pg_stats":[{"pgid":"1.f","version":"38'9722","reported_seq":9951,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.570911+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:22.570911+0000","last_peered":"2024-04-26T02:33:22.570911+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:22.570911+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14761984,"num_objects":1802,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10060","reported_seq":10342,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.482773+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:24.482773+0000","last_peered":"2024-04-26T02:33:24.482773+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:24.482773+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14512652,"num_objects":1772,"num_object_clones":0,"num_object_copies":7088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'9877","reported_seq":10051,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.286910+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:26.286910+0000","last_peered":"2024-04-26T02:33:26.286910+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:26.286910+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9877,"log_dups_size":0,"ondisk_log_size":9877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14358814,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9876,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9714","reported_seq":13744,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:25.926632+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:25.926632+0000","last_peered":"2024-04-26T02:33:25.926632+0000","last_clean":"2024-04-26T02:33:25.926632+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:25.926632+0000","last_undegraded":"2024-04-26T02:33:25.926632+0000","last_fullsized":"2024-04-26T02:33:25.926632+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9714,"log_dups_size":0,"ondisk_log_size":9714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:28.650651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14463500,"num_objects":1766,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"38'9766","reported_seq":9997,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.555296+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:22.555296+0000","last_peered":"2024-04-26T02:33:22.555296+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:22.555296+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14995980,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-98}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"38'9902","reported_seq":10159,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.278954+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:22.278954+0000","last_peered":"2024-04-26T02:33:22.278954+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:22.278954+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13910016,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10033","reported_seq":14290,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:24.404972+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:24.404972+0000","last_peered":"2024-04-26T02:33:24.404972+0000","last_clean":"2024-04-26T02:33:24.404972+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:24.404972+0000","last_undegraded":"2024-04-26T02:33:24.404972+0000","last_fullsized":"2024-04-26T02:33:24.404972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:24.991645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14127628,"num_objects":1725,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10031,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10279","reported_seq":10538,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.339747+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:24.339747+0000","last_peered":"2024-04-26T02:33:24.339747+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:24.339747+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14004748,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'9952","reported_seq":14023,"reported_epoch":38,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.491037+0000","last_change":"2024-04-26T02:30:40.035474+0000","last_active":"2024-04-26T02:33:22.491037+0000","last_peered":"2024-04-26T02:33:22.491037+0000","last_clean":"2024-04-26T02:23:51.181167+0000","last_became_active":"2024-04-26T02:24:59.559898+0000","last_became_peered":"2024-04-26T02:24:59.559898+0000","last_unstale":"2024-04-26T02:33:22.491037+0000","last_undegraded":"2024-04-26T02:24:59.309150+0000","last_fullsized":"2024-04-26T02:24:59.227986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14640152,"num_objects":1788,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2004,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'9925","reported_seq":10103,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.355209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:26.355209+0000","last_peered":"2024-04-26T02:33:26.355209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:26.355209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13423116,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"38'9948","reported_seq":10183,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.404532+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:22.404532+0000","last_peered":"2024-04-26T02:33:22.404532+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:22.404532+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14502674,"num_objects":1771,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"38'9849","reported_seq":10077,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.491069+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:22.491069+0000","last_peered":"2024-04-26T02:33:22.491069+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:22.491069+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14317830,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10048","reported_seq":10270,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.432022+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:26.432022+0000","last_peered":"2024-04-26T02:33:26.432022+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:26.432022+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14178566,"num_objects":1731,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"38'10276","reported_seq":10524,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:22.406626+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:22.406626+0000","last_peered":"2024-04-26T02:33:22.406626+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:22.406626+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15118860,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-133}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10101","reported_seq":14315,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:26.365864+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:26.365864+0000","last_peered":"2024-04-26T02:33:26.365864+0000","last_clean":"2024-04-26T02:33:26.365864+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:26.365864+0000","last_undegraded":"2024-04-26T02:33:26.365864+0000","last_fullsized":"2024-04-26T02:33:26.365864+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14365196,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'10042","reported_seq":10276,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.341557+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:24.341557+0000","last_peered":"2024-04-26T02:33:24.341557+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:24.341557+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14373388,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10040,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":230055104,"num_objects":28090,"num_object_clones":0,"num_object_copies":112360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28090,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159467,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9089,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":457977856,"data_stored":457977856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158994,"ondisk_log_size":158994,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117105,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705564,"kb_used_data":114388,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93666276,"statfs":{"total":96636764160,"available":95914266624,"internally_reserved":0,"allocated":117133312,"data_stored":116929650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,6,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709992,"kb_used_data":111008,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93661848,"statfs":{"total":96636764160,"available":95909732352,"internally_reserved":0,"allocated":113672192,"data_stored":113459149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576680,"kb_used_data":111904,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93795160,"statfs":{"total":96636764160,"available":96046243840,"internally_reserved":0,"allocated":114589696,"data_stored":114376653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712208,"kb_used_data":112772,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93659632,"statfs":{"total":96636764160,"available":95907463168,"internally_reserved":0,"allocated":115478528,"data_stored":115270186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114864128,"data_stored":114864128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113610752,"data_stored":113610752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113311744,"data_stored":113311744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":116191232,"data_stored":116191232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "38'9722", 'reported_seq': 9951, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.570911+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:22.570911+0000', 'last_peered': '2024-04-26T02:33:22.570911+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:22.570911+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9722, 'log_dups_size': 0, 'ondisk_log_size': 9722, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14761984, 'num_objects': 1802, 'num_object_clones': 0, 'num_object_copies': 7208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9722, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "39'10060", 'reported_seq': 10342, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:24.482773+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:33:24.482773+0000', 'last_peered': '2024-04-26T02:33:24.482773+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:24.482773+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14512652, 'num_objects': 1772, 'num_object_clones': 0, 'num_object_copies': 7088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1978, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10058, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'9877", 'reported_seq': 10051, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:26.286910+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:26.286910+0000', 'last_peered': '2024-04-26T02:33:26.286910+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:26.286910+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9877, 'log_dups_size': 0, 'ondisk_log_size': 9877, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14358814, 'num_objects': 1754, 'num_object_clones': 0, 'num_object_copies': 7016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1754, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9876, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -230}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:26.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "38'9766", 'reported_seq': 9997, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.555296+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:22.555296+0000', 'last_peered': '2024-04-26T02:33:22.555296+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:22.555296+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9766, 'log_dups_size': 0, 'ondisk_log_size': 9766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14995980, 'num_objects': 1831, 'num_object_clones': 0, 'num_object_copies': 7324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9764, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -98}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "38'9902", 'reported_seq': 10159, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.278954+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:22.278954+0000', 'last_peered': '2024-04-26T02:33:22.278954+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:22.278954+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9902, 'log_dups_size': 0, 'ondisk_log_size': 9902, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13910016, 'num_objects': 1698, 'num_object_clones': 0, 'num_object_copies': 6792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9902, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -252}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "39'10279", 'reported_seq': 10538, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:24.339747+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:24.339747+0000', 'last_peered': '2024-04-26T02:33:24.339747+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:24.339747+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14004748, 'num_objects': 1710, 'num_object_clones': 0, 'num_object_copies': 6840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10277, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "38'9952", 'reported_seq': 14023, 'reported_epoch': 38, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.491037+0000', 'last_change': '2024-04-26T02:30:40.035474+0000', 'last_active': '2024-04-26T02:33:22.491037+0000', 'last_peered': '2024-04-26T02:33:22.491037+0000', 'last_clean': '2024-04-26T02:23:51.181167+0000', 'last_became_active': '2024-04-26T02:24:59.559898+0000', 'last_became_peered': '2024-04-26T02:24:59.559898+0000', 'last_unstale': '2024-04-26T02:33:22.491037+0000', 'last_undegraded': '2024-04-26T02:24:59.309150+0000', 'last_fullsized': '2024-04-26T02:24:59.227986+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9952, 'log_dups_size': 0, 'ondisk_log_size': 9952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14640152, 'num_objects': 1788, 'num_object_clones': 0, 'num_object_copies': 7152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9949, 'num_write_kb': 46945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2004, 'num_bytes_recovered': 425984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'9925", 'reported_seq': 10103, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:26.355209+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:26.355209+0000', 'last_peered': '2024-04-26T02:33:26.355209+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:26.355209+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9925, 'log_dups_size': 0, 'ondisk_log_size': 9925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13423116, 'num_objects': 1639, 'num_object_clones': 0, 'num_object_copies': 6556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9923, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -356}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "38'9948", 'reported_seq': 10183, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.404532+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:22.404532+0000', 'last_peered': '2024-04-26T02:33:22.404532+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:22.404532+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9948, 'log_dups_size': 0, 'ondisk_log_size': 9948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14502674, 'num_objects': 1771, 'num_object_clones': 0, 'num_object_copies': 7084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9945, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -163}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "38'9849", 'reported_seq': 10077, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.491069+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:22.491069+0000', 'last_peered': '2024-04-26T02:33:22.491069+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:22.491069+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9849, 'log_dups_size': 0, 'ondisk_log_size': 9849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14317830, 'num_objects': 1748, 'num_object_clones': 0, 'num_object_copies': 6992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10048", 'reported_seq': 10270, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:26.432022+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:26.432022+0000', 'last_peered': '2024-04-26T02:33:26.432022+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:26.432022+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14178566, 'num_objects': 1731, 'num_object_clones': 0, 'num_object_copies': 6924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10047, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "38'10276", 'reported_seq': 10524, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:22.406626+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:22.406626+0000', 'last_peered': '2024-04-26T02:33:22.406626+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:22.406626+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15118860, 'num_objects': 1846, 'num_object_clones': 0, 'num_object_copies': 7384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10274, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:26.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "39'10042", 'reported_seq': 10276, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:24.341557+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:24.341557+0000', 'last_peered': '2024-04-26T02:33:24.341557+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:24.341557+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14373388, 'num_objects': 1755, 'num_object_clones': 0, 'num_object_copies': 7020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10040, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:26.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:28.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:28.667 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23680/112360 objects degraded (21.075%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":605,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28090,"data_bytes":230055104,"bytes_used":2769350656,"bytes_avail":383777705984,"bytes_total":386547056640,"degraded_objects":23680,"degraded_total":112360,"degraded_ratio":0.21075115699537203,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:33:28.725 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:28.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:29.954 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:29.954 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:30.030 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-26T02:33:28.704758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226598080,"num_objects":27668,"num_object_clones":0,"num_object_copies":110672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159889,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9095,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159316,"ondisk_log_size":159316,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704872,"kb_used_data":446336,"kb_used_omap":14,"kb_used_meta":2258481,"kb_avail":374782488,"statfs":{"total":386547056640,"available":383777267712,"internally_reserved":0,"allocated":457048064,"data_stored":456200593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2312685107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,11,6,16,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13524992,"num_objects":-1651,"num_object_clones":0,"num_object_copies":-6604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002218"},"pg_stats":[{"pgid":"1.f","version":"40'9782","reported_seq":10015,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.434534+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:27.434534+0000","last_peered":"2024-04-26T02:33:27.434534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:27.434534+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14270464,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"39'10060","reported_seq":10342,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.482773+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:24.482773+0000","last_peered":"2024-04-26T02:33:24.482773+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:24.482773+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14512652,"num_objects":1772,"num_object_clones":0,"num_object_copies":7088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'9877","reported_seq":10051,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.286910+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:26.286910+0000","last_peered":"2024-04-26T02:33:26.286910+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:26.286910+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9877,"log_dups_size":0,"ondisk_log_size":9877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14358814,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9876,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9714","reported_seq":13744,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:25.926632+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:25.926632+0000","last_peered":"2024-04-26T02:33:25.926632+0000","last_clean":"2024-04-26T02:33:25.926632+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:25.926632+0000","last_undegraded":"2024-04-26T02:33:25.926632+0000","last_fullsized":"2024-04-26T02:33:25.926632+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9714,"log_dups_size":0,"ondisk_log_size":9714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:28.650651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14463500,"num_objects":1766,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9817","reported_seq":10049,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.594425+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:27.594425+0000","last_peered":"2024-04-26T02:33:27.594425+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:27.594425+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14578188,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'9959","reported_seq":10220,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.555290+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:27.555290+0000","last_peered":"2024-04-26T02:33:27.555290+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:27.555290+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9959,"log_dups_size":0,"ondisk_log_size":9959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13443072,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9959,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10033","reported_seq":14290,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:24.404972+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:24.404972+0000","last_peered":"2024-04-26T02:33:24.404972+0000","last_clean":"2024-04-26T02:33:24.404972+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:24.404972+0000","last_undegraded":"2024-04-26T02:33:24.404972+0000","last_fullsized":"2024-04-26T02:33:24.404972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:24.991645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14127628,"num_objects":1725,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10031,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'10279","reported_seq":10538,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.339747+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:24.339747+0000","last_peered":"2024-04-26T02:33:24.339747+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:24.339747+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14004748,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10016","reported_seq":14111,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:27.687217+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:27.687217+0000","last_peered":"2024-04-26T02:33:27.687217+0000","last_clean":"2024-04-26T02:33:27.687217+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:27.687217+0000","last_undegraded":"2024-04-26T02:33:27.687217+0000","last_fullsized":"2024-04-26T02:33:27.687217+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:29.968865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14115864,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'9925","reported_seq":10103,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.355209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:26.355209+0000","last_peered":"2024-04-26T02:33:26.355209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:26.355209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13423116,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10016","reported_seq":10255,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.537601+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:27.537601+0000","last_peered":"2024-04-26T02:33:27.537601+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:27.537601+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13945618,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9906","reported_seq":10140,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.612398+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:27.612398+0000","last_peered":"2024-04-26T02:33:27.612398+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:27.612398+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9906,"log_dups_size":0,"ondisk_log_size":9906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13850886,"num_objects":1691,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9905,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10048","reported_seq":10270,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:26.432022+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:26.432022+0000","last_peered":"2024-04-26T02:33:26.432022+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:26.432022+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14178566,"num_objects":1731,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10341","reported_seq":10593,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.477697+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:27.477697+0000","last_peered":"2024-04-26T02:33:27.477697+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:27.477697+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14586380,"num_objects":1781,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10101","reported_seq":14315,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:26.365864+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:26.365864+0000","last_peered":"2024-04-26T02:33:26.365864+0000","last_clean":"2024-04-26T02:33:26.365864+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:26.365864+0000","last_undegraded":"2024-04-26T02:33:26.365864+0000","last_fullsized":"2024-04-26T02:33:26.365864+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14365196,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"39'10042","reported_seq":10276,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:24.341557+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:24.341557+0000","last_peered":"2024-04-26T02:33:24.341557+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:24.341557+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14373388,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10040,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226598080,"num_objects":27668,"num_object_clones":0,"num_object_copies":110672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159889,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9095,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454172672,"data_stored":454172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159316,"ondisk_log_size":159316,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705992,"kb_used_data":110652,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93665848,"statfs":{"total":96636764160,"available":95913828352,"internally_reserved":0,"allocated":113307648,"data_stored":113094605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,10,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709992,"kb_used_data":111008,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93661848,"statfs":{"total":96636764160,"available":95909732352,"internally_reserved":0,"allocated":113672192,"data_stored":113459149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576680,"kb_used_data":111904,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93795160,"statfs":{"total":96636764160,"available":96046243840,"internally_reserved":0,"allocated":114589696,"data_stored":114376653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712208,"kb_used_data":112772,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93659632,"statfs":{"total":96636764160,"available":95907463168,"internally_reserved":0,"allocated":115478528,"data_stored":115270186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114864128,"data_stored":114864128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113610752,"data_stored":113610752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113311744,"data_stored":113311744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112386048,"data_stored":112386048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:33.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:34.560 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:34.561 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:34.640 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-26T02:33:32.705665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222338240,"num_objects":27148,"num_object_clones":0,"num_object_copies":108592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160409,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9138,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159436,"ondisk_log_size":159436,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676952,"kb_used_data":435052,"kb_used_omap":14,"kb_used_meta":2241841,"kb_avail":374810408,"statfs":{"total":386547056640,"available":383805857792,"internally_reserved":0,"allocated":445493248,"data_stored":444641076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2295645747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,9,20,13,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12050432,"num_objects":-1471,"num_object_clones":0,"num_object_copies":-5884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.371871"},"pg_stats":[{"pgid":"1.f","version":"40'9782","reported_seq":10015,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.434534+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:27.434534+0000","last_peered":"2024-04-26T02:33:27.434534+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:27.434534+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14270464,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10126","reported_seq":10494,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:29.514769+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:29.514769+0000","last_peered":"2024-04-26T02:33:29.514769+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:29.514769+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13971980,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10124,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-229}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'9929","reported_seq":10103,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.275231+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:31.275231+0000","last_peered":"2024-04-26T02:33:31.275231+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:31.275231+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9929,"log_dups_size":0,"ondisk_log_size":9929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13932830,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9928,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9778","reported_seq":13808,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:31.368535+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:31.368535+0000","last_peered":"2024-04-26T02:33:31.368535+0000","last_clean":"2024-04-26T02:33:31.368535+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:31.368535+0000","last_undegraded":"2024-04-26T02:33:31.368535+0000","last_fullsized":"2024-04-26T02:33:31.368535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9778,"log_dups_size":0,"ondisk_log_size":9778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:34.587300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939212,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9776,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9817","reported_seq":10049,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.594425+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:27.594425+0000","last_peered":"2024-04-26T02:33:27.594425+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:27.594425+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14578188,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'9959","reported_seq":10220,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.555290+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:27.555290+0000","last_peered":"2024-04-26T02:33:27.555290+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:27.555290+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9959,"log_dups_size":0,"ondisk_log_size":9959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13443072,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9959,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10086","reported_seq":14343,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:29.438647+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:29.438647+0000","last_peered":"2024-04-26T02:33:29.438647+0000","last_clean":"2024-04-26T02:33:29.438647+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:29.438647+0000","last_undegraded":"2024-04-26T02:33:29.438647+0000","last_fullsized":"2024-04-26T02:33:29.438647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:30.988481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13693452,"num_objects":1672,"num_object_clones":0,"num_object_copies":6688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10084,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10345","reported_seq":10604,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:29.534140+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:29.534140+0000","last_peered":"2024-04-26T02:33:29.534140+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:29.534140+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13464076,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10016","reported_seq":14111,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:27.687217+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:27.687217+0000","last_peered":"2024-04-26T02:33:27.687217+0000","last_clean":"2024-04-26T02:33:27.687217+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:27.687217+0000","last_undegraded":"2024-04-26T02:33:27.687217+0000","last_fullsized":"2024-04-26T02:33:27.687217+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:29.968865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14115864,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'9973","reported_seq":10151,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.365859+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:31.365859+0000","last_peered":"2024-04-26T02:33:31.365859+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:31.365859+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13029900,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10016","reported_seq":10255,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.537601+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:27.537601+0000","last_peered":"2024-04-26T02:33:27.537601+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:27.537601+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13945618,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9906","reported_seq":10140,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.612398+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:27.612398+0000","last_peered":"2024-04-26T02:33:27.612398+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:27.612398+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9906,"log_dups_size":0,"ondisk_log_size":9906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13850886,"num_objects":1691,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9905,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10101","reported_seq":10323,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.391583+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:31.391583+0000","last_peered":"2024-04-26T02:33:31.391583+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:31.391583+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13744390,"num_objects":1678,"num_object_clones":0,"num_object_copies":6712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-367}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10341","reported_seq":10593,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:27.477697+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:27.477697+0000","last_peered":"2024-04-26T02:33:27.477697+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:27.477697+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14586380,"num_objects":1781,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10152","reported_seq":14367,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:31.455447+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:31.455447+0000","last_peered":"2024-04-26T02:33:31.455447+0000","last_clean":"2024-04-26T02:33:31.455447+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:31.455447+0000","last_undegraded":"2024-04-26T02:33:31.455447+0000","last_fullsized":"2024-04-26T02:33:31.455447+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.597937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13947404,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10109","reported_seq":10343,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:29.325617+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:29.325617+0000","last_peered":"2024-04-26T02:33:29.325617+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:29.325617+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13824524,"num_objects":1688,"num_object_clones":0,"num_object_copies":6752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":222338240,"num_objects":27148,"num_object_clones":0,"num_object_copies":108592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160409,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9138,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442396672,"data_stored":442396672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159436,"ondisk_log_size":159436,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705992,"kb_used_data":110652,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93665848,"statfs":{"total":96636764160,"available":95913828352,"internally_reserved":0,"allocated":113307648,"data_stored":113094605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,10,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693708,"kb_used_data":107196,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93678132,"statfs":{"total":96636764160,"available":95926407168,"internally_reserved":0,"allocated":109768704,"data_stored":109555661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581176,"kb_used_data":108080,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93790664,"statfs":{"total":96636764160,"available":96041639936,"internally_reserved":0,"allocated":110673920,"data_stored":110460877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696076,"kb_used_data":109124,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93675764,"statfs":{"total":96636764160,"available":95923982336,"internally_reserved":0,"allocated":111742976,"data_stored":111529933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,13,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110944256,"data_stored":110944256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109416448,"data_stored":109416448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112386048,"data_stored":112386048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:34.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:35.986 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:35.987 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:36.100 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-26T02:33:34.706179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216800448,"num_objects":26472,"num_object_clones":0,"num_object_copies":105888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9195,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159912,"ondisk_log_size":159912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2677712,"kb_used_data":427488,"kb_used_omap":14,"kb_used_meta":2250161,"kb_avail":374809648,"statfs":{"total":386547056640,"available":383805079552,"internally_reserved":0,"allocated":437747712,"data_stored":436895540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2304165427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,8,11,8,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17588224,"num_objects":-2147,"num_object_clones":0,"num_object_copies":-8588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.376638"},"pg_stats":[{"pgid":"1.f","version":"40'9851","reported_seq":10087,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.779649+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:32.779649+0000","last_peered":"2024-04-26T02:33:32.779649+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:32.779649+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13705216,"num_objects":1673,"num_object_clones":0,"num_object_copies":6692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10177","reported_seq":10657,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.618065+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:34.618065+0000","last_peered":"2024-04-26T02:33:34.618065+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:34.618065+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13554188,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'9929","reported_seq":10103,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.275231+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:31.275231+0000","last_peered":"2024-04-26T02:33:31.275231+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:31.275231+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9929,"log_dups_size":0,"ondisk_log_size":9929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13932830,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9928,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9778","reported_seq":13808,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:31.368535+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:31.368535+0000","last_peered":"2024-04-26T02:33:31.368535+0000","last_clean":"2024-04-26T02:33:31.368535+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:31.368535+0000","last_undegraded":"2024-04-26T02:33:31.368535+0000","last_fullsized":"2024-04-26T02:33:31.368535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9778,"log_dups_size":0,"ondisk_log_size":9778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:34.587300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939212,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9776,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9876","reported_seq":10111,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.658138+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:32.658138+0000","last_peered":"2024-04-26T02:33:32.658138+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:32.658138+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14094860,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9874,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10017","reported_seq":10278,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.344160+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:32.344160+0000","last_peered":"2024-04-26T02:33:32.344160+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:32.344160+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12967936,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10147","reported_seq":14399,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:34.411537+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:34.411537+0000","last_peered":"2024-04-26T02:33:34.411537+0000","last_clean":"2024-04-26T02:33:34.411537+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:34.411537+0000","last_undegraded":"2024-04-26T02:33:34.411537+0000","last_fullsized":"2024-04-26T02:33:34.411537+0000","mapping_epoch":33,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:35.989410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13193740,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10145,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10398","reported_seq":10657,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.540060+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:34.540060+0000","last_peered":"2024-04-26T02:33:34.540060+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:34.540060+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13029900,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10396,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10074","reported_seq":14169,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:32.397256+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:32.397256+0000","last_peered":"2024-04-26T02:33:32.397256+0000","last_clean":"2024-04-26T02:33:32.397256+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:32.397256+0000","last_undegraded":"2024-04-26T02:33:32.397256+0000","last_fullsized":"2024-04-26T02:33:32.397256+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:35.890585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13640728,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'9973","reported_seq":10151,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.365859+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:31.365859+0000","last_peered":"2024-04-26T02:33:31.365859+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:31.365859+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9973,"log_dups_size":0,"ondisk_log_size":9973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13029900,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9971,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10081","reported_seq":10320,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.664177+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:32.664177+0000","last_peered":"2024-04-26T02:33:32.664177+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:32.664177+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13413138,"num_objects":1638,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'9968","reported_seq":10201,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.735093+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:32.735093+0000","last_peered":"2024-04-26T02:33:32.735093+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:32.735093+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13342982,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10101","reported_seq":10323,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:31.391583+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:31.391583+0000","last_peered":"2024-04-26T02:33:31.391583+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:31.391583+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13744390,"num_objects":1678,"num_object_clones":0,"num_object_copies":6712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-367}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10415","reported_seq":10667,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:32.625236+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:32.625236+0000","last_peered":"2024-04-26T02:33:32.625236+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:32.625236+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980172,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10152","reported_seq":14367,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:31.455447+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:31.455447+0000","last_peered":"2024-04-26T02:33:31.455447+0000","last_clean":"2024-04-26T02:33:31.455447+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:31.455447+0000","last_undegraded":"2024-04-26T02:33:31.455447+0000","last_fullsized":"2024-04-26T02:33:31.455447+0000","mapping_epoch":35,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:31.597937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13947404,"num_objects":1703,"num_object_clones":0,"num_object_copies":6812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10175","reported_seq":10409,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.374703+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:34.374703+0000","last_peered":"2024-04-26T02:33:34.374703+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:34.374703+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13283852,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":216800448,"num_objects":26472,"num_object_clones":0,"num_object_copies":105888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9195,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434348032,"data_stored":434348032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159912,"ondisk_log_size":159912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706428,"kb_used_data":106924,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93665412,"statfs":{"total":96636764160,"available":95913381888,"internally_reserved":0,"allocated":109490176,"data_stored":109277133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693708,"kb_used_data":107196,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93678132,"statfs":{"total":96636764160,"available":95926407168,"internally_reserved":0,"allocated":109768704,"data_stored":109555661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581176,"kb_used_data":108080,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93790664,"statfs":{"total":96636764160,"available":96041639936,"internally_reserved":0,"allocated":110673920,"data_stored":110460877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696400,"kb_used_data":105288,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93675440,"statfs":{"total":96636764160,"available":95923650560,"internally_reserved":0,"allocated":107814912,"data_stored":107601869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106913792,"data_stored":106913792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109416448,"data_stored":109416448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108367872,"data_stored":108367872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:36.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'9851", 'reported_seq': 10087, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.779649+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:32.779649+0000', 'last_peered': '2024-04-26T02:33:32.779649+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:32.779649+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9851, 'log_dups_size': 0, 'ondisk_log_size': 9851, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13705216, 'num_objects': 1673, 'num_object_clones': 0, 'num_object_copies': 6692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9851, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10177", 'reported_seq': 10657, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:34.618065+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:33:34.618065+0000', 'last_peered': '2024-04-26T02:33:34.618065+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:34.618065+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13554188, 'num_objects': 1655, 'num_object_clones': 0, 'num_object_copies': 6620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1878, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10175, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 202, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'9929", 'reported_seq': 10103, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:31.275231+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:31.275231+0000', 'last_peered': '2024-04-26T02:33:31.275231+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:31.275231+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9929, 'log_dups_size': 0, 'ondisk_log_size': 9929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13932830, 'num_objects': 1702, 'num_object_clones': 0, 'num_object_copies': 6808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1702, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9928, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -282}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'9876", 'reported_seq': 10111, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.658138+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:32.658138+0000', 'last_peered': '2024-04-26T02:33:32.658138+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:32.658138+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9876, 'log_dups_size': 0, 'ondisk_log_size': 9876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14094860, 'num_objects': 1721, 'num_object_clones': 0, 'num_object_copies': 6884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9874, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -208}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:36.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10017", 'reported_seq': 10278, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.344160+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:32.344160+0000', 'last_peered': '2024-04-26T02:33:32.344160+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:32.344160+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12967936, 'num_objects': 1583, 'num_object_clones': 0, 'num_object_copies': 6332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10017, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10398", 'reported_seq': 10657, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:34.540060+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:34.540060+0000', 'last_peered': '2024-04-26T02:33:34.540060+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:34.540060+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13029900, 'num_objects': 1591, 'num_object_clones': 0, 'num_object_copies': 6364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10396, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'9973", 'reported_seq': 10151, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:31.365859+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:31.365859+0000', 'last_peered': '2024-04-26T02:33:31.365859+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:31.365859+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9973, 'log_dups_size': 0, 'ondisk_log_size': 9973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13029900, 'num_objects': 1591, 'num_object_clones': 0, 'num_object_copies': 6364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9971, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10081", 'reported_seq': 10320, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.664177+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:32.664177+0000', 'last_peered': '2024-04-26T02:33:32.664177+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:32.664177+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13413138, 'num_objects': 1638, 'num_object_clones': 0, 'num_object_copies': 6552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10078, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -296}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'9968", 'reported_seq': 10201, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.735093+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:32.735093+0000', 'last_peered': '2024-04-26T02:33:32.735093+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:32.735093+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9968, 'log_dups_size': 0, 'ondisk_log_size': 9968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13342982, 'num_objects': 1629, 'num_object_clones': 0, 'num_object_copies': 6516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9967, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:36.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10101", 'reported_seq': 10323, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:31.391583+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:31.391583+0000', 'last_peered': '2024-04-26T02:33:31.391583+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:31.391583+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13744390, 'num_objects': 1678, 'num_object_clones': 0, 'num_object_copies': 6712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10100, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:36.104 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:36.104 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'10415", 'reported_seq': 10667, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:32.625236+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:32.625236+0000', 'last_peered': '2024-04-26T02:33:32.625236+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:32.625236+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13980172, 'num_objects': 1707, 'num_object_clones': 0, 'num_object_copies': 6828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10413, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:36.104 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:36.104 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10175", 'reported_seq': 10409, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:34.374703+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:34.374703+0000', 'last_peered': '2024-04-26T02:33:34.374703+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:34.374703+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13283852, 'num_objects': 1622, 'num_object_clones': 0, 'num_object_copies': 6488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10173, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:36.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:37.874 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:37.874 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23574/104632 objects degraded (22.530%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":614,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26158,"data_bytes":214228160,"bytes_used":2726281216,"bytes_avail":383820775424,"bytes_total":386547056640,"degraded_objects":23574,"degraded_total":104632,"degraded_ratio":0.22530392231822005,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-26T02:32:40.675089+0000","services":{}},"progress_events":{}} 2024-04-26T02:33:38.006 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:38.006 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:39.595 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:39.596 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:39.687 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-26T02:33:38.707232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210787520,"num_objects":25738,"num_object_clones":0,"num_object_copies":102952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161819,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9195,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160346,"ondisk_log_size":160346,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2663408,"kb_used_data":415948,"kb_used_omap":14,"kb_used_meta":2247409,"kb_avail":374823952,"statfs":{"total":386547056640,"available":383819726848,"internally_reserved":0,"allocated":425930752,"data_stored":425078580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2301347379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,11,9,9,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002974"},"pg_stats":[{"pgid":"1.f","version":"40'9917","reported_seq":10153,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.787671+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:37.787671+0000","last_peered":"2024-04-26T02:33:37.787671+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:37.787671+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13164544,"num_objects":1607,"num_object_clones":0,"num_object_copies":6428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10177","reported_seq":10657,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.618065+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:34.618065+0000","last_peered":"2024-04-26T02:33:34.618065+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:34.618065+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13554188,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'9986","reported_seq":10160,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:36.317634+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:36.317634+0000","last_peered":"2024-04-26T02:33:36.317634+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:36.317634+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13465886,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9985,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9847","reported_seq":13877,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:36.440330+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:36.440330+0000","last_peered":"2024-04-26T02:33:36.440330+0000","last_clean":"2024-04-26T02:33:36.440330+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:36.440330+0000","last_undegraded":"2024-04-26T02:33:36.440330+0000","last_fullsized":"2024-04-26T02:33:36.440330+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9847,"log_dups_size":0,"ondisk_log_size":9847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:39.596211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373964,"num_objects":1633,"num_object_clones":0,"num_object_copies":6532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9845,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9928","reported_seq":10163,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.619696+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:37.619696+0000","last_peered":"2024-04-26T02:33:37.619696+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:37.619696+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9928,"log_dups_size":0,"ondisk_log_size":9928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13668876,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10076","reported_seq":10337,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.816057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:37.816057+0000","last_peered":"2024-04-26T02:33:37.816057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:37.816057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12484608,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10147","reported_seq":14399,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:34.411537+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:34.411537+0000","last_peered":"2024-04-26T02:33:34.411537+0000","last_clean":"2024-04-26T02:33:34.411537+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:34.411537+0000","last_undegraded":"2024-04-26T02:33:34.411537+0000","last_fullsized":"2024-04-26T02:33:34.411537+0000","mapping_epoch":33,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:35.989410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13193740,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10145,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10398","reported_seq":10657,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.540060+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:34.540060+0000","last_peered":"2024-04-26T02:33:34.540060+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:34.540060+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13029900,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10396,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10129","reported_seq":14224,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:37.741923+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:37.741923+0000","last_peered":"2024-04-26T02:33:37.741923+0000","last_clean":"2024-04-26T02:33:37.741923+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:37.741923+0000","last_undegraded":"2024-04-26T02:33:37.741923+0000","last_fullsized":"2024-04-26T02:33:37.741923+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:41.805782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13190168,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10035","reported_seq":10213,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:36.399721+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:36.399721+0000","last_peered":"2024-04-26T02:33:36.399721+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:36.399721+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12521996,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10137","reported_seq":10376,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.751962+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:37.751962+0000","last_peered":"2024-04-26T02:33:37.751962+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:37.751962+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12954386,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10045","reported_seq":10279,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.578701+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:37.578701+0000","last_peered":"2024-04-26T02:33:37.578701+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:37.578701+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12712198,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10157","reported_seq":10379,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:36.355085+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:36.355085+0000","last_peered":"2024-04-26T02:33:36.355085+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:36.355085+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13285638,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10156,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10470","reported_seq":10722,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.722582+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:37.722582+0000","last_peered":"2024-04-26T02:33:37.722582+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:37.722582+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13529612,"num_objects":1652,"num_object_clones":0,"num_object_copies":6608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10468,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10222","reported_seq":14437,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:36.476684+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:36.476684+0000","last_peered":"2024-04-26T02:33:36.476684+0000","last_clean":"2024-04-26T02:33:36.476684+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:36.476684+0000","last_undegraded":"2024-04-26T02:33:36.476684+0000","last_fullsized":"2024-04-26T02:33:36.476684+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:36.627044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13373964,"num_objects":1633,"num_object_clones":0,"num_object_copies":6532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10175","reported_seq":10409,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:34.374703+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:34.374703+0000","last_peered":"2024-04-26T02:33:34.374703+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:34.374703+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13283852,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":210787520,"num_objects":25738,"num_object_clones":0,"num_object_copies":102952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161819,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9195,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":422424576,"data_stored":422424576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160346,"ondisk_log_size":160346,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707452,"kb_used_data":102388,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93664388,"statfs":{"total":96636764160,"available":95912333312,"internally_reserved":0,"allocated":104845312,"data_stored":104632269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694728,"kb_used_data":104064,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93677112,"statfs":{"total":96636764160,"available":95925362688,"internally_reserved":0,"allocated":106561536,"data_stored":106348493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564828,"kb_used_data":104208,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93807012,"statfs":{"total":96636764160,"available":96058380288,"internally_reserved":0,"allocated":106708992,"data_stored":106495949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696400,"kb_used_data":105288,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93675440,"statfs":{"total":96636764160,"available":95923650560,"internally_reserved":0,"allocated":107814912,"data_stored":107601869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106913792,"data_stored":106913792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105652224,"data_stored":105652224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105435136,"data_stored":105435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":104423424,"data_stored":104423424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:42.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:44.345 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:44.346 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:44.435 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-26T02:33:42.708320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":206191808,"num_objects":25177,"num_object_clones":0,"num_object_copies":100708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162380,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9241,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160407,"ondisk_log_size":160407,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703036,"kb_used_data":402836,"kb_used_omap":14,"kb_used_meta":2300145,"kb_avail":374784324,"statfs":{"total":386547056640,"available":383779147776,"internally_reserved":0,"allocated":412504064,"data_stored":411651892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2355349043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,6,12,8,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18341888,"num_objects":-2239,"num_object_clones":0,"num_object_copies":-8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003163"},"pg_stats":[{"pgid":"1.f","version":"40'9917","reported_seq":10153,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.787671+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:37.787671+0000","last_peered":"2024-04-26T02:33:37.787671+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:37.787671+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13164544,"num_objects":1607,"num_object_clones":0,"num_object_copies":6428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10243","reported_seq":10817,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.661911+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:39.661911+0000","last_peered":"2024-04-26T02:33:39.661911+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:39.661911+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13013516,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10048","reported_seq":10222,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.632688+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:41.632688+0000","last_peered":"2024-04-26T02:33:41.632688+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:41.632688+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12957982,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10047,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9925","reported_seq":13955,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:41.624878+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:41.624878+0000","last_peered":"2024-04-26T02:33:41.624878+0000","last_clean":"2024-04-26T02:33:41.624878+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:41.624878+0000","last_undegraded":"2024-04-26T02:33:41.624878+0000","last_fullsized":"2024-04-26T02:33:41.624878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:44.653237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12734988,"num_objects":1555,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9928","reported_seq":10163,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.619696+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:37.619696+0000","last_peered":"2024-04-26T02:33:37.619696+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:37.619696+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9928,"log_dups_size":0,"ondisk_log_size":9928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13668876,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10076","reported_seq":10337,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.816057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:37.816057+0000","last_peered":"2024-04-26T02:33:37.816057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:37.816057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12484608,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-426}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10211","reported_seq":14468,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:39.603894+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:39.603894+0000","last_peered":"2024-04-26T02:33:39.603894+0000","last_clean":"2024-04-26T02:33:39.603894+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:39.603894+0000","last_undegraded":"2024-04-26T02:33:39.603894+0000","last_fullsized":"2024-04-26T02:33:39.603894+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:42.011926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12669452,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10209,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10463","reported_seq":10722,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.678744+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:39.678744+0000","last_peered":"2024-04-26T02:33:39.678744+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:39.678744+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12497420,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10129","reported_seq":14224,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:37.741923+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:37.741923+0000","last_peered":"2024-04-26T02:33:37.741923+0000","last_clean":"2024-04-26T02:33:37.741923+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:37.741923+0000","last_undegraded":"2024-04-26T02:33:37.741923+0000","last_fullsized":"2024-04-26T02:33:37.741923+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:41.805782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13190168,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10094","reported_seq":10272,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.462648+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:41.462648+0000","last_peered":"2024-04-26T02:33:41.462648+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:41.462648+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12038668,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10092,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10137","reported_seq":10376,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.751962+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:37.751962+0000","last_peered":"2024-04-26T02:33:37.751962+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:37.751962+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12954386,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10045","reported_seq":10279,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.578701+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:37.578701+0000","last_peered":"2024-04-26T02:33:37.578701+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:37.578701+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12712198,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10211","reported_seq":10433,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.365963+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:41.365963+0000","last_peered":"2024-04-26T02:33:41.365963+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:41.365963+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12843270,"num_objects":1568,"num_object_clones":0,"num_object_copies":6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10470","reported_seq":10722,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:37.722582+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:37.722582+0000","last_peered":"2024-04-26T02:33:37.722582+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:37.722582+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13529612,"num_objects":1652,"num_object_clones":0,"num_object_copies":6608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10468,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10277","reported_seq":14491,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:41.548419+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:41.548419+0000","last_peered":"2024-04-26T02:33:41.548419+0000","last_clean":"2024-04-26T02:33:41.548419+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:41.548419+0000","last_undegraded":"2024-04-26T02:33:41.548419+0000","last_fullsized":"2024-04-26T02:33:41.548419+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:46.677247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12923404,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10275,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10233","reported_seq":10467,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.686311+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:39.686311+0000","last_peered":"2024-04-26T02:33:39.686311+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:39.686311+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12808716,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":206191808,"num_objects":25177,"num_object_clones":0,"num_object_copies":100708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162380,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9241,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":410378240,"data_stored":410378240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160407,"ondisk_log_size":160407,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707452,"kb_used_data":102388,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93664388,"statfs":{"total":96636764160,"available":95912333312,"internally_reserved":0,"allocated":104845312,"data_stored":104632269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716412,"kb_used_data":99380,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93655428,"statfs":{"total":96636764160,"available":95903158272,"internally_reserved":0,"allocated":101765120,"data_stored":101552077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564336,"kb_used_data":99552,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93807504,"statfs":{"total":96636764160,"available":96058884096,"internally_reserved":0,"allocated":101941248,"data_stored":101728205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714836,"kb_used_data":101516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93657004,"statfs":{"total":96636764160,"available":95904772096,"internally_reserved":0,"allocated":103952384,"data_stored":103739341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102989824,"data_stored":102989824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101556224,"data_stored":101556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101408768,"data_stored":101408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":104423424,"data_stored":104423424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:44.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:46.124 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:46.124 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:46.233 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-04-26T02:33:44.708795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202906816,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162781,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9241,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2703256,"kb_used_data":398896,"kb_used_omap":14,"kb_used_meta":2304305,"kb_avail":374784104,"statfs":{"total":386547056640,"available":383778922496,"internally_reserved":0,"allocated":408469504,"data_stored":407617332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2359608883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,6,14,13,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19431424,"num_objects":-2372,"num_object_clones":0,"num_object_copies":-9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003129"},"pg_stats":[{"pgid":"1.f","version":"40'9975","reported_seq":10212,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.762243+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:42.762243+0000","last_peered":"2024-04-26T02:33:42.762243+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:42.762243+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9975,"log_dups_size":0,"ondisk_log_size":9975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12689408,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10243","reported_seq":10817,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.661911+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:39.661911+0000","last_peered":"2024-04-26T02:33:39.661911+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:39.661911+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13013516,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10048","reported_seq":10222,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.632688+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:41.632688+0000","last_peered":"2024-04-26T02:33:41.632688+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:41.632688+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12957982,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10047,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9925","reported_seq":13955,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:41.624878+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:41.624878+0000","last_peered":"2024-04-26T02:33:41.624878+0000","last_clean":"2024-04-26T02:33:41.624878+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:41.624878+0000","last_undegraded":"2024-04-26T02:33:41.624878+0000","last_fullsized":"2024-04-26T02:33:41.624878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:44.653237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12734988,"num_objects":1555,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'9987","reported_seq":10221,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.778536+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:42.778536+0000","last_peered":"2024-04-26T02:33:42.778536+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:42.778536+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13185548,"num_objects":1610,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-319}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10138","reported_seq":10399,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.936629+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:42.936629+0000","last_peered":"2024-04-26T02:33:42.936629+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:42.936629+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11976704,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10138,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10211","reported_seq":14468,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:39.603894+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:39.603894+0000","last_peered":"2024-04-26T02:33:39.603894+0000","last_clean":"2024-04-26T02:33:39.603894+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:39.603894+0000","last_undegraded":"2024-04-26T02:33:39.603894+0000","last_fullsized":"2024-04-26T02:33:39.603894+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:42.011926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12669452,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10209,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10463","reported_seq":10722,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.678744+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:39.678744+0000","last_peered":"2024-04-26T02:33:39.678744+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:39.678744+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12497420,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10191","reported_seq":14286,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:42.828269+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:42.828269+0000","last_peered":"2024-04-26T02:33:42.828269+0000","last_clean":"2024-04-26T02:33:42.828269+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:42.828269+0000","last_undegraded":"2024-04-26T02:33:42.828269+0000","last_fullsized":"2024-04-26T02:33:42.828269+0000","mapping_epoch":39,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:47.819924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12682264,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10094","reported_seq":10272,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.462648+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:41.462648+0000","last_peered":"2024-04-26T02:33:41.462648+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:41.462648+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12038668,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10092,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10178","reported_seq":10417,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.567137+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:42.567137+0000","last_peered":"2024-04-26T02:33:42.567137+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:42.567137+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12618514,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10103","reported_seq":10337,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.797151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:42.797151+0000","last_peered":"2024-04-26T02:33:42.797151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:42.797151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12237062,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-404}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10211","reported_seq":10433,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:41.365963+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:41.365963+0000","last_peered":"2024-04-26T02:33:41.365963+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:41.365963+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12843270,"num_objects":1568,"num_object_clones":0,"num_object_copies":6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10531","reported_seq":10783,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:42.849258+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:42.849258+0000","last_peered":"2024-04-26T02:33:42.849258+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:42.849258+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13029900,"num_objects":1591,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-388}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10277","reported_seq":14491,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:41.548419+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:41.548419+0000","last_peered":"2024-04-26T02:33:41.548419+0000","last_clean":"2024-04-26T02:33:41.548419+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:41.548419+0000","last_undegraded":"2024-04-26T02:33:41.548419+0000","last_fullsized":"2024-04-26T02:33:41.548419+0000","mapping_epoch":35,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:46.677247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12923404,"num_objects":1578,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10275,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10233","reported_seq":10467,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:39.686311+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:39.686311+0000","last_peered":"2024-04-26T02:33:39.686311+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:39.686311+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12808716,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202906816,"num_objects":24776,"num_object_clones":0,"num_object_copies":99104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162781,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9241,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406224896,"data_stored":406224896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707672,"kb_used_data":98448,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93664168,"statfs":{"total":96636764160,"available":95912108032,"internally_reserved":0,"allocated":100810752,"data_stored":100597709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716412,"kb_used_data":99380,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93655428,"statfs":{"total":96636764160,"available":95903158272,"internally_reserved":0,"allocated":101765120,"data_stored":101552077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564336,"kb_used_data":99552,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93807504,"statfs":{"total":96636764160,"available":96058884096,"internally_reserved":0,"allocated":101941248,"data_stored":101728205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714836,"kb_used_data":101516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93657004,"statfs":{"total":96636764160,"available":95904772096,"internally_reserved":0,"allocated":103952384,"data_stored":103739341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102989824,"data_stored":102989824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101556224,"data_stored":101556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101408768,"data_stored":101408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100270080,"data_stored":100270080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'9975", 'reported_seq': 10212, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.762243+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:42.762243+0000', 'last_peered': '2024-04-26T02:33:42.762243+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:42.762243+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9975, 'log_dups_size': 0, 'ondisk_log_size': 9975, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12689408, 'num_objects': 1549, 'num_object_clones': 0, 'num_object_copies': 6196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9975, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -378}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10243", 'reported_seq': 10817, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:39.661911+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:33:39.661911+0000', 'last_peered': '2024-04-26T02:33:39.661911+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:39.661911+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13013516, 'num_objects': 1589, 'num_object_clones': 0, 'num_object_copies': 6356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1832, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10241, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 248, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:46.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10048", 'reported_seq': 10222, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:41.632688+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:41.632688+0000', 'last_peered': '2024-04-26T02:33:41.632688+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:41.632688+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12957982, 'num_objects': 1583, 'num_object_clones': 0, 'num_object_copies': 6332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1583, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10047, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -401}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'9987", 'reported_seq': 10221, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.778536+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:42.778536+0000', 'last_peered': '2024-04-26T02:33:42.778536+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:42.778536+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 9987, 'log_dups_size': 0, 'ondisk_log_size': 9987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13185548, 'num_objects': 1610, 'num_object_clones': 0, 'num_object_copies': 6440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9985, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -319}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10138", 'reported_seq': 10399, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.936629+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:42.936629+0000', 'last_peered': '2024-04-26T02:33:42.936629+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:42.936629+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11976704, 'num_objects': 1462, 'num_object_clones': 0, 'num_object_copies': 5848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10138, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -488}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10463", 'reported_seq': 10722, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:39.678744+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:39.678744+0000', 'last_peered': '2024-04-26T02:33:39.678744+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:39.678744+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12497420, 'num_objects': 1526, 'num_object_clones': 0, 'num_object_copies': 6104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10461, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10094", 'reported_seq': 10272, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:41.462648+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:41.462648+0000', 'last_peered': '2024-04-26T02:33:41.462648+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:41.462648+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12038668, 'num_objects': 1470, 'num_object_clones': 0, 'num_object_copies': 5880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10092, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -525}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:46.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10178", 'reported_seq': 10417, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.567137+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:42.567137+0000', 'last_peered': '2024-04-26T02:33:42.567137+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:42.567137+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12618514, 'num_objects': 1541, 'num_object_clones': 0, 'num_object_copies': 6164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10175, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10103", 'reported_seq': 10337, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.797151+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:42.797151+0000', 'last_peered': '2024-04-26T02:33:42.797151+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:42.797151+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12237062, 'num_objects': 1494, 'num_object_clones': 0, 'num_object_copies': 5976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10102, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10211", 'reported_seq': 10433, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:41.365963+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:41.365963+0000', 'last_peered': '2024-04-26T02:33:41.365963+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:41.365963+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12843270, 'num_objects': 1568, 'num_object_clones': 0, 'num_object_copies': 6272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10210, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'10531", 'reported_seq': 10783, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:42.849258+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:42.849258+0000', 'last_peered': '2024-04-26T02:33:42.849258+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:42.849258+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13029900, 'num_objects': 1591, 'num_object_clones': 0, 'num_object_copies': 6364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10529, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:46.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10233", 'reported_seq': 10467, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:39.686311+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:39.686311+0000', 'last_peered': '2024-04-26T02:33:39.686311+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:39.686311+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12808716, 'num_objects': 1564, 'num_object_clones': 0, 'num_object_copies': 6256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10231, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:46.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:48.514 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23459/98048 objects degraded (23.926%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24512,"data_bytes":200744128,"bytes_used":2768375808,"bytes_avail":383778680832,"bytes_total":386547056640,"degraded_objects":23459,"degraded_total":98048,"degraded_ratio":0.23926036227154046,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":163},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-04-26T02:33:44.708879+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:33:48.652 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:48.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:50.408 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:50.409 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:50.529 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-26T02:33:48.709842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194600128,"num_objects":23762,"num_object_clones":0,"num_object_copies":95048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9310,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2691908,"kb_used_data":383388,"kb_used_omap":14,"kb_used_meta":2308465,"kb_avail":374795452,"statfs":{"total":386547056640,"available":383790542848,"internally_reserved":0,"allocated":392589312,"data_stored":391737140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2363868723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,5,10,20,17,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19628032,"num_objects":-2396,"num_object_clones":0,"num_object_copies":-9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003096"},"pg_stats":[{"pgid":"1.f","version":"40'10043","reported_seq":10280,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:48.080803+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:48.080803+0000","last_peered":"2024-04-26T02:33:48.080803+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:48.080803+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12132352,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10309","reported_seq":11020,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:44.848808+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:44.848808+0000","last_peered":"2024-04-26T02:33:44.848808+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:44.848808+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12472844,"num_objects":1523,"num_object_clones":0,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10111","reported_seq":10285,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:46.457166+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:46.457166+0000","last_peered":"2024-04-26T02:33:46.457166+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:46.457166+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12441886,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10110,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'9990","reported_seq":14020,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:46.730314+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:46.730314+0000","last_peered":"2024-04-26T02:33:46.730314+0000","last_clean":"2024-04-26T02:33:46.730314+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:46.730314+0000","last_undegraded":"2024-04-26T02:33:46.730314+0000","last_fullsized":"2024-04-26T02:33:46.730314+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:49.746524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12202508,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9988,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10069","reported_seq":10303,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:48.099669+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:48.099669+0000","last_peered":"2024-04-26T02:33:48.099669+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:48.099669+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12513804,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10191","reported_seq":10452,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:47.820922+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:47.820922+0000","last_peered":"2024-04-26T02:33:47.820922+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:47.820922+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11542528,"num_objects":1409,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10262","reported_seq":14519,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:44.619926+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:44.619926+0000","last_peered":"2024-04-26T02:33:44.619926+0000","last_clean":"2024-04-26T02:33:44.619926+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:44.619926+0000","last_undegraded":"2024-04-26T02:33:44.619926+0000","last_fullsized":"2024-04-26T02:33:44.619926+0000","mapping_epoch":33,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:47.963335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12251660,"num_objects":1496,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10534","reported_seq":10793,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:44.643274+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:44.643274+0000","last_peered":"2024-04-26T02:33:44.643274+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:44.643274+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11915788,"num_objects":1455,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-597}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10241","reported_seq":14335,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:48.071956+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:48.071956+0000","last_peered":"2024-04-26T02:33:48.071956+0000","last_clean":"2024-04-26T02:33:48.071956+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:48.071956+0000","last_undegraded":"2024-04-26T02:33:48.071956+0000","last_fullsized":"2024-04-26T02:33:48.071956+0000","mapping_epoch":39,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12272664,"num_objects":1499,"num_object_clones":0,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10154","reported_seq":10332,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:46.500186+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:46.500186+0000","last_peered":"2024-04-26T02:33:46.500186+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:46.500186+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547148,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-585}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10236","reported_seq":10475,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:48.134421+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:48.134421+0000","last_peered":"2024-04-26T02:33:48.134421+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:48.134421+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12143378,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10233,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10158","reported_seq":10391,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:48.004692+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:48.004692+0000","last_peered":"2024-04-26T02:33:48.004692+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:48.004692+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11786502,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10157,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-459}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10269","reported_seq":10491,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:46.601887+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:46.601887+0000","last_peered":"2024-04-26T02:33:46.601887+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:46.601887+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12368134,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10268,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-535}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10590","reported_seq":10842,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:48.018157+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:48.018157+0000","last_peered":"2024-04-26T02:33:48.018157+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:48.018157+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12546572,"num_objects":1532,"num_object_clones":0,"num_object_copies":6128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-447}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10356","reported_seq":14571,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:46.754569+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:46.754569+0000","last_peered":"2024-04-26T02:33:46.754569+0000","last_clean":"2024-04-26T02:33:46.754569+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:46.754569+0000","last_undegraded":"2024-04-26T02:33:46.754569+0000","last_fullsized":"2024-04-26T02:33:46.754569+0000","mapping_epoch":35,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:51.777130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12276236,"num_objects":1499,"num_object_clones":0,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10309","reported_seq":10543,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:44.780144+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:44.780144+0000","last_peered":"2024-04-26T02:33:44.780144+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:44.780144+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12186124,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194600128,"num_objects":23762,"num_object_clones":0,"num_object_copies":95048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9310,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389984256,"data_stored":389984256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117110,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712152,"kb_used_data":94600,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93659688,"statfs":{"total":96636764160,"available":95907520512,"internally_reserved":0,"allocated":96870400,"data_stored":96657357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,2,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700144,"kb_used_data":95592,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93671696,"statfs":{"total":96636764160,"available":95919816704,"internally_reserved":0,"allocated":97886208,"data_stored":97673165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,1,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564540,"kb_used_data":95604,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93807300,"statfs":{"total":96636764160,"available":96058675200,"internally_reserved":0,"allocated":97898496,"data_stored":97685453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,6,14,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715072,"kb_used_data":97592,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93656768,"statfs":{"total":96636764160,"available":95904530432,"internally_reserved":0,"allocated":99934208,"data_stored":99721165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98856960,"data_stored":98856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97497088,"data_stored":97497088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97378304,"data_stored":97378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96251904,"data_stored":96251904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:53.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:55.257 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:55.259 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:55.398 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-26T02:33:54.711321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":186408128,"num_objects":22762,"num_object_clones":0,"num_object_copies":91048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9350,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160622,"ondisk_log_size":160622,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685524,"kb_used_data":367248,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374801836,"statfs":{"total":386547056640,"available":383797080064,"internally_reserved":0,"allocated":376061952,"data_stored":375209780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2373830195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,1,5,14,17,12,12,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19783680,"num_objects":-2415,"num_object_clones":0,"num_object_copies":-9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.f","version":"40'10106","reported_seq":10343,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.108085+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:53.108085+0000","last_peered":"2024-04-26T02:33:53.108085+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:53.108085+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10370","reported_seq":11162,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.897996+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:49.897996+0000","last_peered":"2024-04-26T02:33:49.897996+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:49.897996+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11973132,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10167","reported_seq":10341,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.514825+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:51.514825+0000","last_peered":"2024-04-26T02:33:51.514825+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:51.514825+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11983134,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10166,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10052","reported_seq":14082,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:51.916766+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:51.916766+0000","last_peered":"2024-04-26T02:33:51.916766+0000","last_clean":"2024-04-26T02:33:51.916766+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:51.916766+0000","last_undegraded":"2024-04-26T02:33:51.916766+0000","last_fullsized":"2024-04-26T02:33:51.916766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:54.845350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11694604,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10127","reported_seq":10362,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.171450+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:53.171450+0000","last_peered":"2024-04-26T02:33:53.171450+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:53.171450+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12038668,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10257","reported_seq":10513,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.279409+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:53.279409+0000","last_peered":"2024-04-26T02:33:53.279409+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:53.279409+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11001856,"num_objects":1343,"num_object_clones":0,"num_object_copies":5372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10257,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10336","reported_seq":14592,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:49.903855+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:49.903855+0000","last_peered":"2024-04-26T02:33:49.903855+0000","last_clean":"2024-04-26T02:33:49.903855+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:49.903855+0000","last_undegraded":"2024-04-26T02:33:49.903855+0000","last_fullsized":"2024-04-26T02:33:49.903855+0000","mapping_epoch":33,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:53.912921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11645452,"num_objects":1422,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10579","reported_seq":10838,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.287529+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:49.287529+0000","last_peered":"2024-04-26T02:33:49.287529+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:49.287529+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547148,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10319","reported_seq":14414,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:53.207424+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:53.207424+0000","last_peered":"2024-04-26T02:33:53.207424+0000","last_clean":"2024-04-26T02:33:53.207424+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:53.207424+0000","last_undegraded":"2024-04-26T02:33:53.207424+0000","last_fullsized":"2024-04-26T02:33:53.207424+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:53.844768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11633688,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10215","reported_seq":10393,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.618686+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:51.618686+0000","last_peered":"2024-04-26T02:33:51.618686+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:51.618686+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11047436,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10302","reported_seq":10540,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.174209+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:53.174209+0000","last_peered":"2024-04-26T02:33:53.174209+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:53.174209+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11602706,"num_objects":1417,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10213","reported_seq":10447,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:52.810924+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:52.810924+0000","last_peered":"2024-04-26T02:33:52.810924+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:52.810924+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11335942,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-514}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10326","reported_seq":10548,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.632857+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:51.632857+0000","last_peered":"2024-04-26T02:33:51.632857+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:51.632857+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11901190,"num_objects":1453,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10670","reported_seq":10922,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.089994+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:53.089994+0000","last_peered":"2024-04-26T02:33:53.089994+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:53.089994+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891212,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-527}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10405","reported_seq":14620,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:51.828383+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:51.828383+0000","last_peered":"2024-04-26T02:33:51.828383+0000","last_clean":"2024-04-26T02:33:51.828383+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:51.828383+0000","last_undegraded":"2024-04-26T02:33:51.828383+0000","last_fullsized":"2024-04-26T02:33:51.828383+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11874828,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10378","reported_seq":10612,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.670755+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:49.670755+0000","last_peered":"2024-04-26T02:33:49.670755+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:49.670755+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11620876,"num_objects":1419,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":186408128,"num_objects":22762,"num_object_clones":0,"num_object_copies":91048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9350,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374013952,"data_stored":374013952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160622,"ondisk_log_size":160622,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695832,"kb_used_data":90752,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93676008,"statfs":{"total":96636764160,"available":95924232192,"internally_reserved":0,"allocated":92930048,"data_stored":92717005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,0,0,6,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700492,"kb_used_data":91764,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93671348,"statfs":{"total":96636764160,"available":95919460352,"internally_reserved":0,"allocated":93966336,"data_stored":93753293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587000,"kb_used_data":91692,"kb_used_omap":4,"kb_used_meta":495291,"kb_avail":93784840,"statfs":{"total":96636764160,"available":96035676160,"internally_reserved":0,"allocated":93892608,"data_stored":93679565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":507178703},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":93040,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":95272960,"data_stored":95059917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,4,7,1,4,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94928896,"data_stored":94928896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93478912,"data_stored":93478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93413376,"data_stored":93413376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92192768,"data_stored":92192768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:55.399 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:33:56.612 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:56.612 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:33:56.710 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-26T02:33:54.711321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":186408128,"num_objects":22762,"num_object_clones":0,"num_object_copies":91048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9350,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160622,"ondisk_log_size":160622,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685524,"kb_used_data":367248,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374801836,"statfs":{"total":386547056640,"available":383797080064,"internally_reserved":0,"allocated":376061952,"data_stored":375209780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2373830195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,1,5,14,17,12,12,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19783680,"num_objects":-2415,"num_object_clones":0,"num_object_copies":-9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.f","version":"40'10106","reported_seq":10343,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.108085+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:53.108085+0000","last_peered":"2024-04-26T02:33:53.108085+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:53.108085+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10370","reported_seq":11162,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.897996+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:49.897996+0000","last_peered":"2024-04-26T02:33:49.897996+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:49.897996+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11973132,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10167","reported_seq":10341,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.514825+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:51.514825+0000","last_peered":"2024-04-26T02:33:51.514825+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:51.514825+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11983134,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10166,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10052","reported_seq":14082,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:51.916766+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:51.916766+0000","last_peered":"2024-04-26T02:33:51.916766+0000","last_clean":"2024-04-26T02:33:51.916766+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:51.916766+0000","last_undegraded":"2024-04-26T02:33:51.916766+0000","last_fullsized":"2024-04-26T02:33:51.916766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:54.845350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11694604,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10127","reported_seq":10362,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.171450+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:53.171450+0000","last_peered":"2024-04-26T02:33:53.171450+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:53.171450+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12038668,"num_objects":1470,"num_object_clones":0,"num_object_copies":5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-459}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10257","reported_seq":10513,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.279409+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:53.279409+0000","last_peered":"2024-04-26T02:33:53.279409+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:53.279409+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11001856,"num_objects":1343,"num_object_clones":0,"num_object_copies":5372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10257,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10336","reported_seq":14592,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:49.903855+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:49.903855+0000","last_peered":"2024-04-26T02:33:49.903855+0000","last_clean":"2024-04-26T02:33:49.903855+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:49.903855+0000","last_undegraded":"2024-04-26T02:33:49.903855+0000","last_fullsized":"2024-04-26T02:33:49.903855+0000","mapping_epoch":33,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:53.912921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11645452,"num_objects":1422,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10579","reported_seq":10838,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.287529+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:49.287529+0000","last_peered":"2024-04-26T02:33:49.287529+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:49.287529+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11547148,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10319","reported_seq":14414,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:53.207424+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:53.207424+0000","last_peered":"2024-04-26T02:33:53.207424+0000","last_clean":"2024-04-26T02:33:53.207424+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:53.207424+0000","last_undegraded":"2024-04-26T02:33:53.207424+0000","last_fullsized":"2024-04-26T02:33:53.207424+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:53.844768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11633688,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10215","reported_seq":10393,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.618686+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:51.618686+0000","last_peered":"2024-04-26T02:33:51.618686+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:51.618686+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11047436,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10302","reported_seq":10540,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.174209+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:53.174209+0000","last_peered":"2024-04-26T02:33:53.174209+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:53.174209+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11602706,"num_objects":1417,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10213","reported_seq":10447,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:52.810924+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:52.810924+0000","last_peered":"2024-04-26T02:33:52.810924+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:52.810924+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11335942,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-514}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10326","reported_seq":10548,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:51.632857+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:51.632857+0000","last_peered":"2024-04-26T02:33:51.632857+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:51.632857+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11901190,"num_objects":1453,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10670","reported_seq":10922,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:53.089994+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:53.089994+0000","last_peered":"2024-04-26T02:33:53.089994+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:53.089994+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891212,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-527}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10405","reported_seq":14620,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:51.828383+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:51.828383+0000","last_peered":"2024-04-26T02:33:51.828383+0000","last_clean":"2024-04-26T02:33:51.828383+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:51.828383+0000","last_undegraded":"2024-04-26T02:33:51.828383+0000","last_fullsized":"2024-04-26T02:33:51.828383+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11874828,"num_objects":1450,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10378","reported_seq":10612,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:49.670755+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:49.670755+0000","last_peered":"2024-04-26T02:33:49.670755+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:49.670755+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11620876,"num_objects":1419,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":186408128,"num_objects":22762,"num_object_clones":0,"num_object_copies":91048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164795,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9350,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374013952,"data_stored":374013952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160622,"ondisk_log_size":160622,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695832,"kb_used_data":90752,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93676008,"statfs":{"total":96636764160,"available":95924232192,"internally_reserved":0,"allocated":92930048,"data_stored":92717005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,0,0,6,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700492,"kb_used_data":91764,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93671348,"statfs":{"total":96636764160,"available":95919460352,"internally_reserved":0,"allocated":93966336,"data_stored":93753293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587000,"kb_used_data":91692,"kb_used_omap":4,"kb_used_meta":495291,"kb_avail":93784840,"statfs":{"total":96636764160,"available":96035676160,"internally_reserved":0,"allocated":93892608,"data_stored":93679565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":507178703},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702200,"kb_used_data":93040,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93669640,"statfs":{"total":96636764160,"available":95917711360,"internally_reserved":0,"allocated":95272960,"data_stored":95059917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,4,7,1,4,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94928896,"data_stored":94928896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93478912,"data_stored":93478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93413376,"data_stored":93413376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92192768,"data_stored":92192768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:33:56.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:33:56.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10106", 'reported_seq': 10343, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:53.108085+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:33:53.108085+0000', 'last_peered': '2024-04-26T02:33:53.108085+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:33:53.108085+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11616256, 'num_objects': 1418, 'num_object_clones': 0, 'num_object_copies': 5672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10106, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10370", 'reported_seq': 11162, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:49.897996+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:33:49.897996+0000', 'last_peered': '2024-04-26T02:33:49.897996+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:33:49.897996+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11973132, 'num_objects': 1462, 'num_object_clones': 0, 'num_object_copies': 5848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1723, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10368, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 357, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10167", 'reported_seq': 10341, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:51.514825+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:33:51.514825+0000', 'last_peered': '2024-04-26T02:33:51.514825+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:33:51.514825+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11983134, 'num_objects': 1464, 'num_object_clones': 0, 'num_object_copies': 5856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1464, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10166, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -520}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10127", 'reported_seq': 10362, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:53.171450+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:33:53.171450+0000', 'last_peered': '2024-04-26T02:33:53.171450+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:33:53.171450+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'100", 'ondisk_log_start': "28'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12038668, 'num_objects': 1470, 'num_object_clones': 0, 'num_object_copies': 5880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10125, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -459}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10257", 'reported_seq': 10513, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:53.279409+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:33:53.279409+0000', 'last_peered': '2024-04-26T02:33:53.279409+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:33:53.279409+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11001856, 'num_objects': 1343, 'num_object_clones': 0, 'num_object_copies': 5372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10257, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10579", 'reported_seq': 10838, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:49.287529+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:33:49.287529+0000', 'last_peered': '2024-04-26T02:33:49.287529+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:33:49.287529+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11547148, 'num_objects': 1410, 'num_object_clones': 0, 'num_object_copies': 5640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10577, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:56.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10215", 'reported_seq': 10393, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:51.618686+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:33:51.618686+0000', 'last_peered': '2024-04-26T02:33:51.618686+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:33:51.618686+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11047436, 'num_objects': 1349, 'num_object_clones': 0, 'num_object_copies': 5396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10213, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -646}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10302", 'reported_seq': 10540, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:53.174209+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:33:53.174209+0000', 'last_peered': '2024-04-26T02:33:53.174209+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:33:53.174209+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11602706, 'num_objects': 1417, 'num_object_clones': 0, 'num_object_copies': 5668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10299, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10213", 'reported_seq': 10447, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:52.810924+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:33:52.810924+0000', 'last_peered': '2024-04-26T02:33:52.810924+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:33:52.810924+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11335942, 'num_objects': 1384, 'num_object_clones': 0, 'num_object_copies': 5536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10212, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -514}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10326", 'reported_seq': 10548, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:51.632857+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:33:51.632857+0000', 'last_peered': '2024-04-26T02:33:51.632857+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:33:51.632857+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11901190, 'num_objects': 1453, 'num_object_clones': 0, 'num_object_copies': 5812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10325, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'10670", 'reported_seq': 10922, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:53.089994+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:33:53.089994+0000', 'last_peered': '2024-04-26T02:33:53.089994+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:33:53.089994+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11891212, 'num_objects': 1452, 'num_object_clones': 0, 'num_object_copies': 5808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10668, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:33:56.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10378", 'reported_seq': 10612, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:33:49.670755+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:33:49.670755+0000', 'last_peered': '2024-04-26T02:33:49.670755+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:33:49.670755+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11620876, 'num_objects': 1419, 'num_object_clones': 0, 'num_object_copies': 5676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10376, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:33:56.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:33:58.631 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:33:58.631 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23373/90060 objects degraded (25.953%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":635,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22515,"data_bytes":184384704,"bytes_used":2750201856,"bytes_avail":383796854784,"bytes_total":386547056640,"degraded_objects":23373,"degraded_total":90060,"degraded_ratio":0.25952698201199198,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-26T02:33:56.711889+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:33:58.673 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:33:58.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:00.297 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:00.298 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:00.422 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-04-26T02:33:58.712307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":178568384,"num_objects":21805,"num_object_clones":0,"num_object_copies":87220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165752,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9396,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2669924,"kb_used_data":351620,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374817436,"statfs":{"total":386547056640,"available":383813054464,"internally_reserved":0,"allocated":360058880,"data_stored":359206708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2373830195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,11,8,14,11,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22175744,"num_objects":-2707,"num_object_clones":0,"num_object_copies":-10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2707,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"1.f","version":"40'10159","reported_seq":10396,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:57.800123+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:33:57.800123+0000","last_peered":"2024-04-26T02:33:57.800123+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:33:57.800123+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-562}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10441","reported_seq":11324,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:54.959292+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:54.959292+0000","last_peered":"2024-04-26T02:33:54.959292+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:54.959292+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11391500,"num_objects":1391,"num_object_clones":0,"num_object_copies":5564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10225","reported_seq":10399,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:56.784575+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:33:56.784575+0000","last_peered":"2024-04-26T02:33:56.784575+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:33:56.784575+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11507998,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10224,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-578}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10109","reported_seq":14139,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:56.635845+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:33:56.635845+0000","last_peered":"2024-04-26T02:33:56.635845+0000","last_clean":"2024-04-26T02:33:56.635845+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:33:56.635845+0000","last_undegraded":"2024-04-26T02:33:56.635845+0000","last_fullsized":"2024-04-26T02:33:56.635845+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:00.013200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227660,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10188","reported_seq":10423,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:58.145430+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:33:58.145430+0000","last_peered":"2024-04-26T02:33:58.145430+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:33:58.145430+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11538956,"num_objects":1409,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10186,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10321","reported_seq":10581,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:58.315633+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:33:58.315633+0000","last_peered":"2024-04-26T02:33:58.315633+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:33:58.315633+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10477568,"num_objects":1279,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10321,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10389","reported_seq":14646,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:54.946742+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:54.946742+0000","last_peered":"2024-04-26T02:33:54.946742+0000","last_clean":"2024-04-26T02:33:54.946742+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:54.946742+0000","last_undegraded":"2024-04-26T02:33:54.946742+0000","last_fullsized":"2024-04-26T02:33:54.946742+0000","mapping_epoch":33,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:58.914141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11211276,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10639","reported_seq":10898,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:54.806752+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:54.806752+0000","last_peered":"2024-04-26T02:33:54.806752+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:54.806752+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11055628,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10369","reported_seq":14464,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:58.312508+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:33:58.312508+0000","last_peered":"2024-04-26T02:33:58.312508+0000","last_clean":"2024-04-26T02:33:58.312508+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:33:58.312508+0000","last_undegraded":"2024-04-26T02:33:58.312508+0000","last_fullsized":"2024-04-26T02:33:58.312508+0000","mapping_epoch":39,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:59.741247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11224088,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10271","reported_seq":10449,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:56.782886+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:33:56.782886+0000","last_peered":"2024-04-26T02:33:56.782886+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:33:56.782886+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10588684,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10373","reported_seq":10612,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:58.290858+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:33:58.290858+0000","last_peered":"2024-04-26T02:33:58.290858+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:33:58.290858+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11021074,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10282","reported_seq":10516,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:58.322386+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:33:58.322386+0000","last_peered":"2024-04-26T02:33:58.322386+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:33:58.322386+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10770694,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10281,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10391","reported_seq":10612,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:56.807284+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:33:56.807284+0000","last_peered":"2024-04-26T02:33:56.807284+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:33:56.807284+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11368710,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10724","reported_seq":10975,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:58.290957+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:33:58.290957+0000","last_peered":"2024-04-26T02:33:58.290957+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:33:58.290957+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11448844,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10457","reported_seq":14672,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:56.640870+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:33:56.640870+0000","last_peered":"2024-04-26T02:33:56.640870+0000","last_clean":"2024-04-26T02:33:56.640870+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:33:56.640870+0000","last_undegraded":"2024-04-26T02:33:56.640870+0000","last_fullsized":"2024-04-26T02:33:56.640870+0000","mapping_epoch":35,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:33:56.861463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11448844,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10455,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10441","reported_seq":10675,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:54.977500+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:54.977500+0000","last_peered":"2024-04-26T02:33:54.977500+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:54.977500+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11104780,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":178568384,"num_objects":21805,"num_object_clones":0,"num_object_copies":87220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165752,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9396,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":357986304,"data_stored":357986304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117112,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696228,"kb_used_data":86948,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93675612,"statfs":{"total":96636764160,"available":95923826688,"internally_reserved":0,"allocated":89034752,"data_stored":88821709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,4,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700744,"kb_used_data":87868,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93671096,"statfs":{"total":96636764160,"available":95919202304,"internally_reserved":0,"allocated":89976832,"data_stored":89763789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570532,"kb_used_data":87704,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93801308,"statfs":{"total":96636764160,"available":96052539392,"internally_reserved":0,"allocated":89808896,"data_stored":89595853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494399183},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,1,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702420,"kb_used_data":89100,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93669420,"statfs":{"total":96636764160,"available":95917486080,"internally_reserved":0,"allocated":91238400,"data_stored":91025357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,3,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90849280,"data_stored":90849280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89473024,"data_stored":89473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89432064,"data_stored":89432064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":88231936,"data_stored":88231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:03.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:04.959 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:05.265 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-04-26T02:34:04.717970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":170654912,"num_objects":20839,"num_object_clones":0,"num_object_copies":83356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166718,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9449,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2671112,"kb_used_data":336216,"kb_used_omap":14,"kb_used_meta":2334833,"kb_avail":374816248,"statfs":{"total":386547056640,"available":383811837952,"internally_reserved":0,"allocated":344285184,"data_stored":343433012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2390869555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,5,11,15,11,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19570688,"num_objects":-2389,"num_object_clones":0,"num_object_copies":-9556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007146"},"pg_stats":[{"pgid":"1.f","version":"40'10213","reported_seq":10450,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:02.999126+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:02.999126+0000","last_peered":"2024-04-26T02:34:02.999126+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:02.999126+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739712,"num_objects":1311,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10490","reported_seq":11480,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:59.825293+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:33:59.825293+0000","last_peered":"2024-04-26T02:33:59.825293+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:33:59.825293+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10990092,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10488,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10296","reported_seq":10468,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.871221+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:01.871221+0000","last_peered":"2024-04-26T02:34:01.871221+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:01.871221+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10926366,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10295,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10164","reported_seq":14194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:01.651344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:01.651344+0000","last_peered":"2024-04-26T02:34:01.651344+0000","last_clean":"2024-04-26T02:34:01.651344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:01.651344+0000","last_undegraded":"2024-04-26T02:34:01.651344+0000","last_fullsized":"2024-04-26T02:34:01.651344+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:05.075821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10777100,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10258","reported_seq":10493,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.196962+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:03.196962+0000","last_peered":"2024-04-26T02:34:03.196962+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:03.196962+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10965516,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10368","reported_seq":10629,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.301748+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:03.301748+0000","last_peered":"2024-04-26T02:34:03.301748+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:03.301748+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":1232,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10444","reported_seq":14701,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:33:59.822559+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:33:59.822559+0000","last_peered":"2024-04-26T02:33:59.822559+0000","last_clean":"2024-04-26T02:33:59.822559+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:33:59.822559+0000","last_undegraded":"2024-04-26T02:33:59.822559+0000","last_fullsized":"2024-04-26T02:33:59.822559+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:03.999563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10760716,"num_objects":1314,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10712","reported_seq":10971,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:59.846740+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:33:59.846740+0000","last_peered":"2024-04-26T02:33:59.846740+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:33:59.846740+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10457612,"num_objects":1277,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10425","reported_seq":14520,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:03.189377+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:03.189377+0000","last_peered":"2024-04-26T02:34:03.189377+0000","last_clean":"2024-04-26T02:34:03.189377+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:03.189377+0000","last_undegraded":"2024-04-26T02:34:03.189377+0000","last_fullsized":"2024-04-26T02:34:03.189377+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:04.854646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10765336,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10326","reported_seq":10504,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.707853+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:01.707853+0000","last_peered":"2024-04-26T02:34:01.707853+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:01.707853+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10138124,"num_objects":1238,"num_object_clones":0,"num_object_copies":4952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10431","reported_seq":10670,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.259513+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:03.259513+0000","last_peered":"2024-04-26T02:34:03.259513+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:03.259513+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10545938,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10353","reported_seq":10587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.219531+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:03.219531+0000","last_peered":"2024-04-26T02:34:03.219531+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:03.219531+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10189062,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-654}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10440","reported_seq":10662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.740426+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:01.740426+0000","last_peered":"2024-04-26T02:34:01.740426+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:01.740426+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10967302,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10804","reported_seq":11052,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.244474+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:03.244474+0000","last_peered":"2024-04-26T02:34:03.244474+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:03.244474+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10793484,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10525","reported_seq":14740,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:01.838555+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:01.838555+0000","last_peered":"2024-04-26T02:34:01.838555+0000","last_clean":"2024-04-26T02:34:01.838555+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:01.838555+0000","last_undegraded":"2024-04-26T02:34:01.838555+0000","last_fullsized":"2024-04-26T02:34:01.838555+0000","mapping_epoch":35,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:02.054605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891788,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10496","reported_seq":10730,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:33:59.844356+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:33:59.844356+0000","last_peered":"2024-04-26T02:33:59.844356+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:33:59.844356+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10654220,"num_objects":1301,"num_object_clones":0,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":170654912,"num_objects":20839,"num_object_clones":0,"num_object_copies":83356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166718,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9449,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":342167552,"data_stored":342167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696452,"kb_used_data":83060,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93675388,"statfs":{"total":96636764160,"available":95923597312,"internally_reserved":0,"allocated":85053440,"data_stored":84840397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701068,"kb_used_data":84028,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93670772,"statfs":{"total":96636764160,"available":95918870528,"internally_reserved":0,"allocated":86044672,"data_stored":85831629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570844,"kb_used_data":83860,"kb_used_omap":4,"kb_used_meta":486971,"kb_avail":93800996,"statfs":{"total":96636764160,"available":96052219904,"internally_reserved":0,"allocated":85872640,"data_stored":85659597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498659023},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,3,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":85268,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":87314432,"data_stored":87101389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,1,3,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86945792,"data_stored":86945792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85520384,"data_stored":85520384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85446656,"data_stored":85446656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84254720,"data_stored":84254720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:05.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:06.774 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:06.775 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:06.841 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-04-26T02:34:06.718555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":168844480,"num_objects":20618,"num_object_clones":0,"num_object_copies":82472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166939,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9507,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2655104,"kb_used_data":332672,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374832256,"statfs":{"total":386547056640,"available":383828230144,"internally_reserved":0,"allocated":340656128,"data_stored":339803956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2378090035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,8,14,10,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17563648,"num_objects":-2144,"num_object_clones":0,"num_object_copies":-8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007234"},"pg_stats":[{"pgid":"1.f","version":"40'10213","reported_seq":10450,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:02.999126+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:02.999126+0000","last_peered":"2024-04-26T02:34:02.999126+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:02.999126+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739712,"num_objects":1311,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10544","reported_seq":11650,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:05.095343+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:05.095343+0000","last_peered":"2024-04-26T02:34:05.095343+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:05.095343+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10547724,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10542,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10296","reported_seq":10468,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.871221+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:01.871221+0000","last_peered":"2024-04-26T02:34:01.871221+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:01.871221+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10926366,"num_objects":1335,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10295,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10164","reported_seq":14194,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:01.651344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:01.651344+0000","last_peered":"2024-04-26T02:34:01.651344+0000","last_clean":"2024-04-26T02:34:01.651344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:01.651344+0000","last_undegraded":"2024-04-26T02:34:01.651344+0000","last_fullsized":"2024-04-26T02:34:01.651344+0000","mapping_epoch":37,"log_start":"28'100","ondisk_log_start":"28'100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:05.075821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10777100,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10258","reported_seq":10493,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.196962+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:03.196962+0000","last_peered":"2024-04-26T02:34:03.196962+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:03.196962+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10965516,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10368","reported_seq":10629,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.301748+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:03.301748+0000","last_peered":"2024-04-26T02:34:03.301748+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:03.301748+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":1232,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10488","reported_seq":14745,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:05.037385+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:05.037385+0000","last_peered":"2024-04-26T02:34:05.037385+0000","last_clean":"2024-04-26T02:34:05.037385+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:05.037385+0000","last_undegraded":"2024-04-26T02:34:05.037385+0000","last_fullsized":"2024-04-26T02:34:05.037385+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:09.962538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10400268,"num_objects":1270,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10773","reported_seq":11032,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:04.916619+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:04.916619+0000","last_peered":"2024-04-26T02:34:04.916619+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:04.916619+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9957900,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10425","reported_seq":14520,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:03.189377+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:03.189377+0000","last_peered":"2024-04-26T02:34:03.189377+0000","last_clean":"2024-04-26T02:34:03.189377+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:03.189377+0000","last_undegraded":"2024-04-26T02:34:03.189377+0000","last_fullsized":"2024-04-26T02:34:03.189377+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:04.854646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10765336,"num_objects":1315,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10326","reported_seq":10504,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.707853+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:01.707853+0000","last_peered":"2024-04-26T02:34:01.707853+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:01.707853+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10138124,"num_objects":1238,"num_object_clones":0,"num_object_copies":4952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10431","reported_seq":10670,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.259513+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:03.259513+0000","last_peered":"2024-04-26T02:34:03.259513+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:03.259513+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10545938,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10353","reported_seq":10587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.219531+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:03.219531+0000","last_peered":"2024-04-26T02:34:03.219531+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:03.219531+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10189062,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-654}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10440","reported_seq":10662,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:01.740426+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:01.740426+0000","last_peered":"2024-04-26T02:34:01.740426+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:01.740426+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10967302,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10804","reported_seq":11052,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:03.244474+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:03.244474+0000","last_peered":"2024-04-26T02:34:03.244474+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:03.244474+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10793484,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10802,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10525","reported_seq":14740,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:01.838555+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:01.838555+0000","last_peered":"2024-04-26T02:34:01.838555+0000","last_clean":"2024-04-26T02:34:01.838555+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:01.838555+0000","last_undegraded":"2024-04-26T02:34:01.838555+0000","last_fullsized":"2024-04-26T02:34:01.838555+0000","mapping_epoch":35,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:02.054605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891788,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10558","reported_seq":10792,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:04.988750+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:04.988750+0000","last_peered":"2024-04-26T02:34:04.988750+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:04.988750+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10146316,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10556,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":168844480,"num_objects":20618,"num_object_clones":0,"num_object_copies":82472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166939,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9507,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":338382848,"data_stored":338382848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696452,"kb_used_data":83060,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93675388,"statfs":{"total":96636764160,"available":95923597312,"internally_reserved":0,"allocated":85053440,"data_stored":84840397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701068,"kb_used_data":84028,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93670772,"statfs":{"total":96636764160,"available":95918870528,"internally_reserved":0,"allocated":86044672,"data_stored":85831629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570844,"kb_used_data":83860,"kb_used_omap":4,"kb_used_meta":486971,"kb_avail":93800996,"statfs":{"total":96636764160,"available":96052219904,"internally_reserved":0,"allocated":85872640,"data_stored":85659597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498659023},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,3,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686740,"kb_used_data":81724,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93685100,"statfs":{"total":96636764160,"available":95933542400,"internally_reserved":0,"allocated":83685376,"data_stored":83472333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619507407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83161088,"data_stored":83161088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85520384,"data_stored":85520384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85446656,"data_stored":85446656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84254720,"data_stored":84254720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10213", 'reported_seq': 10450, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:02.999126+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:02.999126+0000', 'last_peered': '2024-04-26T02:34:02.999126+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:02.999126+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10739712, 'num_objects': 1311, 'num_object_clones': 0, 'num_object_copies': 5244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10213, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10544", 'reported_seq': 11650, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:05.095343+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:05.095343+0000', 'last_peered': '2024-04-26T02:34:05.095343+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:05.095343+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10547724, 'num_objects': 1288, 'num_object_clones': 0, 'num_object_copies': 5152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1566, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10542, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 514, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10296", 'reported_seq': 10468, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:01.871221+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:01.871221+0000', 'last_peered': '2024-04-26T02:34:01.871221+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:01.871221+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10926366, 'num_objects': 1335, 'num_object_clones': 0, 'num_object_copies': 5340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1335, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10295, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -649}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:06.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10258", 'reported_seq': 10493, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:03.196962+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:03.196962+0000', 'last_peered': '2024-04-26T02:34:03.196962+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:03.196962+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'200", 'ondisk_log_start': "28'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10965516, 'num_objects': 1339, 'num_object_clones': 0, 'num_object_copies': 5356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10256, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -590}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10368", 'reported_seq': 10629, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:03.301748+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:03.301748+0000', 'last_peered': '2024-04-26T02:34:03.301748+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:03.301748+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10092544, 'num_objects': 1232, 'num_object_clones': 0, 'num_object_copies': 4928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10368, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -718}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10773", 'reported_seq': 11032, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:04.916619+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:04.916619+0000', 'last_peered': '2024-04-26T02:34:04.916619+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:04.916619+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9957900, 'num_objects': 1216, 'num_object_clones': 0, 'num_object_copies': 4864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10771, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10326", 'reported_seq': 10504, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:01.707853+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:01.707853+0000', 'last_peered': '2024-04-26T02:34:01.707853+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:01.707853+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10138124, 'num_objects': 1238, 'num_object_clones': 0, 'num_object_copies': 4952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10324, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -757}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10431", 'reported_seq': 10670, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:03.259513+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:03.259513+0000', 'last_peered': '2024-04-26T02:34:03.259513+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:03.259513+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10545938, 'num_objects': 1288, 'num_object_clones': 0, 'num_object_copies': 5152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10428, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -646}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:06.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10353", 'reported_seq': 10587, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:03.219531+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:03.219531+0000', 'last_peered': '2024-04-26T02:34:03.219531+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:03.219531+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10189062, 'num_objects': 1244, 'num_object_clones': 0, 'num_object_copies': 4976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10352, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10440", 'reported_seq': 10662, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:01.740426+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:01.740426+0000', 'last_peered': '2024-04-26T02:34:01.740426+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:01.740426+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10967302, 'num_objects': 1339, 'num_object_clones': 0, 'num_object_copies': 5356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10439, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -706}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'10804", 'reported_seq': 11052, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:03.244474+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:03.244474+0000', 'last_peered': '2024-04-26T02:34:03.244474+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:03.244474+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10793484, 'num_objects': 1318, 'num_object_clones': 0, 'num_object_copies': 5272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10802, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:06.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10558", 'reported_seq': 10792, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:04.988750+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:04.988750+0000', 'last_peered': '2024-04-26T02:34:04.988750+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:04.988750+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10146316, 'num_objects': 1239, 'num_object_clones': 0, 'num_object_copies': 4956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10556, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:06.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:08.413 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:08.414 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23262/82472 objects degraded (28.206%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":644,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20618,"data_bytes":168844480,"bytes_used":2718826496,"bytes_avail":383828230144,"bytes_total":386547056640,"degraded_objects":23262,"degraded_total":82472,"degraded_ratio":0.28205936560287126,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-26T02:33:56.711889+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:34:08.468 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:08.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:09.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:09.667 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:09.772 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-04-26T02:34:08.719228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163290304,"num_objects":19940,"num_object_clones":0,"num_object_copies":79760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167617,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9507,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658108,"kb_used_data":321792,"kb_used_omap":14,"kb_used_meta":2336241,"kb_avail":374829252,"statfs":{"total":386547056640,"available":383825154048,"internally_reserved":0,"allocated":329515008,"data_stored":328662836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2392311347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,8,5,11,14,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21094400,"num_objects":-2575,"num_object_clones":0,"num_object_copies":-10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007404"},"pg_stats":[{"pgid":"1.f","version":"40'10283","reported_seq":10518,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.180622+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:08.180622+0000","last_peered":"2024-04-26T02:34:08.180622+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:08.180622+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10166272,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10544","reported_seq":11650,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:05.095343+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:05.095343+0000","last_peered":"2024-04-26T02:34:05.095343+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:05.095343+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10547724,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10542,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10354","reported_seq":10528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:06.830915+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:06.830915+0000","last_peered":"2024-04-26T02:34:06.830915+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:06.830915+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10451230,"num_objects":1277,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10353,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-707}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10216","reported_seq":14246,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:06.993234+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:06.993234+0000","last_peered":"2024-04-26T02:34:06.993234+0000","last_clean":"2024-04-26T02:34:06.993234+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:06.993234+0000","last_undegraded":"2024-04-26T02:34:06.993234+0000","last_fullsized":"2024-04-26T02:34:06.993234+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:10.973412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351116,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10214,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10313","reported_seq":10548,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.150570+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:08.150570+0000","last_peered":"2024-04-26T02:34:08.150570+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:08.150570+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10514956,"num_objects":1284,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10411","reported_seq":10672,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.089496+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:08.089496+0000","last_peered":"2024-04-26T02:34:08.089496+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:08.089496+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9740288,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10488","reported_seq":14745,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:05.037385+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:05.037385+0000","last_peered":"2024-04-26T02:34:05.037385+0000","last_clean":"2024-04-26T02:34:05.037385+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:05.037385+0000","last_undegraded":"2024-04-26T02:34:05.037385+0000","last_fullsized":"2024-04-26T02:34:05.037385+0000","mapping_epoch":33,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:09.962538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10400268,"num_objects":1270,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10773","reported_seq":11032,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:04.916619+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:04.916619+0000","last_peered":"2024-04-26T02:34:04.916619+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:04.916619+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9957900,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10481","reported_seq":14576,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:08.185481+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:08.185481+0000","last_peered":"2024-04-26T02:34:08.185481+0000","last_clean":"2024-04-26T02:34:08.185481+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:08.185481+0000","last_undegraded":"2024-04-26T02:34:08.185481+0000","last_fullsized":"2024-04-26T02:34:08.185481+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:09.889669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10306584,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10478,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10385","reported_seq":10563,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:06.895060+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:06.895060+0000","last_peered":"2024-04-26T02:34:06.895060+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:06.895060+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9654796,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10383,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-816}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10484","reported_seq":10723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:07.735290+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:07.735290+0000","last_peered":"2024-04-26T02:34:07.735290+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:07.735290+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10111762,"num_objects":1235,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10413","reported_seq":10646,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.131522+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:08.131522+0000","last_peered":"2024-04-26T02:34:08.131522+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:08.131522+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9697542,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10412,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-714}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10501","reported_seq":10723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:06.842713+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:06.842713+0000","last_peered":"2024-04-26T02:34:06.842713+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:06.842713+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10467590,"num_objects":1278,"num_object_clones":0,"num_object_copies":5112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10850","reported_seq":11102,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.173250+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:08.173250+0000","last_peered":"2024-04-26T02:34:08.173250+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:08.173250+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10416652,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10590","reported_seq":14805,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:06.917428+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:06.917428+0000","last_peered":"2024-04-26T02:34:06.917428+0000","last_clean":"2024-04-26T02:34:06.917428+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:06.917428+0000","last_undegraded":"2024-04-26T02:34:06.917428+0000","last_fullsized":"2024-04-26T02:34:06.917428+0000","mapping_epoch":35,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:08.026567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10359308,"num_objects":1265,"num_object_clones":0,"num_object_copies":5060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10558","reported_seq":10792,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:04.988750+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:04.988750+0000","last_peered":"2024-04-26T02:34:04.988750+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:04.988750+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10146316,"num_objects":1239,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10556,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":163290304,"num_objects":19940,"num_object_clones":0,"num_object_copies":79760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167617,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9507,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327364608,"data_stored":327364608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697092,"kb_used_data":79548,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93674748,"statfs":{"total":96636764160,"available":95922941952,"internally_reserved":0,"allocated":81457152,"data_stored":81244109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,1,4,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684816,"kb_used_data":80252,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93687024,"statfs":{"total":96636764160,"available":95935512576,"internally_reserved":0,"allocated":82178048,"data_stored":81965005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589460,"kb_used_data":80268,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93782380,"statfs":{"total":96636764160,"available":96033157120,"internally_reserved":0,"allocated":82194432,"data_stored":81981389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,2,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686740,"kb_used_data":81724,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93685100,"statfs":{"total":96636764160,"available":95933542400,"internally_reserved":0,"allocated":83685376,"data_stored":83472333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619507407},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83161088,"data_stored":83161088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81768448,"data_stored":81768448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81707008,"data_stored":81707008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":80728064,"data_stored":80728064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:12.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:14.210 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:14.210 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:14.269 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-04-26T02:34:12.720223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":159227072,"num_objects":19444,"num_object_clones":0,"num_object_copies":77776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168113,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9575,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2642880,"kb_used_data":310752,"kb_used_omap":14,"kb_used_meta":2332081,"kb_avail":374844480,"statfs":{"total":386547056640,"available":383840747520,"internally_reserved":0,"allocated":318210048,"data_stored":317357876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2388051507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,6,11,10,13,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17440768,"num_objects":-2129,"num_object_clones":0,"num_object_copies":-8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2129,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"1.f","version":"40'10283","reported_seq":10518,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.180622+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:08.180622+0000","last_peered":"2024-04-26T02:34:08.180622+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:08.180622+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10166272,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-686}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10599","reported_seq":11841,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:10.150917+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:10.150917+0000","last_peered":"2024-04-26T02:34:10.150917+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:10.150917+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10097164,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10413","reported_seq":10587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:11.904145+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:11.904145+0000","last_peered":"2024-04-26T02:34:11.904145+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:11.904145+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9967902,"num_objects":1218,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10412,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10275","reported_seq":14305,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:12.055541+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:12.055541+0000","last_peered":"2024-04-26T02:34:12.055541+0000","last_clean":"2024-04-26T02:34:12.055541+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:12.055541+0000","last_undegraded":"2024-04-26T02:34:12.055541+0000","last_fullsized":"2024-04-26T02:34:12.055541+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:16.003127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9867788,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10313","reported_seq":10548,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.150570+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:08.150570+0000","last_peered":"2024-04-26T02:34:08.150570+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:08.150570+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10514956,"num_objects":1284,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10411","reported_seq":10672,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.089496+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:08.089496+0000","last_peered":"2024-04-26T02:34:08.089496+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:08.089496+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9740288,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10548","reported_seq":14805,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:10.097498+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:10.097498+0000","last_peered":"2024-04-26T02:34:10.097498+0000","last_clean":"2024-04-26T02:34:10.097498+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:10.097498+0000","last_undegraded":"2024-04-26T02:34:10.097498+0000","last_fullsized":"2024-04-26T02:34:10.097498+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9908748,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10828","reported_seq":11087,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:10.079206+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:10.079206+0000","last_peered":"2024-04-26T02:34:10.079206+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:10.079206+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9507340,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10826,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-891}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10481","reported_seq":14576,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:08.185481+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:08.185481+0000","last_peered":"2024-04-26T02:34:08.185481+0000","last_clean":"2024-04-26T02:34:08.185481+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:08.185481+0000","last_undegraded":"2024-04-26T02:34:08.185481+0000","last_fullsized":"2024-04-26T02:34:08.185481+0000","mapping_epoch":39,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:09.889669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10306584,"num_objects":1259,"num_object_clones":0,"num_object_copies":5036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10478,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10425","reported_seq":10604,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:12.041585+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:12.041585+0000","last_peered":"2024-04-26T02:34:12.041585+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:12.041585+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9327116,"num_objects":1139,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10484","reported_seq":10723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:07.735290+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:07.735290+0000","last_peered":"2024-04-26T02:34:07.735290+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:07.735290+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10111762,"num_objects":1235,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10413","reported_seq":10646,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.131522+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:08.131522+0000","last_peered":"2024-04-26T02:34:08.131522+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:08.131522+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9697542,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10412,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-714}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10560","reported_seq":10782,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:11.999632+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:11.999632+0000","last_peered":"2024-04-26T02:34:11.999632+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:11.999632+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9984262,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10850","reported_seq":11102,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:08.173250+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:08.173250+0000","last_peered":"2024-04-26T02:34:08.173250+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:08.173250+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10416652,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10642","reported_seq":14857,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:12.128354+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:12.128354+0000","last_peered":"2024-04-26T02:34:12.128354+0000","last_clean":"2024-04-26T02:34:12.128354+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:12.128354+0000","last_undegraded":"2024-04-26T02:34:12.128354+0000","last_fullsized":"2024-04-26T02:34:12.128354+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:13.034324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9933324,"num_objects":1213,"num_object_clones":0,"num_object_copies":4852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10615","reported_seq":10849,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:09.994754+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:09.994754+0000","last_peered":"2024-04-26T02:34:09.994754+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:09.994754+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9679372,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":159227072,"num_objects":19444,"num_object_clones":0,"num_object_copies":77776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168113,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9575,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315867136,"data_stored":315867136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697092,"kb_used_data":79548,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93674748,"statfs":{"total":96636764160,"available":95922941952,"internally_reserved":0,"allocated":81457152,"data_stored":81244109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,1,4,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685276,"kb_used_data":76564,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93686564,"statfs":{"total":96636764160,"available":95935041536,"internally_reserved":0,"allocated":78401536,"data_stored":78188493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573300,"kb_used_data":76588,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93798540,"statfs":{"total":96636764160,"available":96049704960,"internally_reserved":0,"allocated":78426112,"data_stored":78213069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687212,"kb_used_data":78052,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93684628,"statfs":{"total":96636764160,"available":95933059072,"internally_reserved":0,"allocated":79925248,"data_stored":79712205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79376384,"data_stored":79376384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77930496,"data_stored":77930496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77832192,"data_stored":77832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":80728064,"data_stored":80728064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:14.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:16.000 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:16.000 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:16.080 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-04-26T02:34:14.720707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":155417792,"num_objects":18979,"num_object_clones":0,"num_object_copies":75916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168578,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9575,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626608,"kb_used_data":306952,"kb_used_omap":14,"kb_used_meta":2319601,"kb_avail":374860752,"statfs":{"total":386547056640,"available":383857410048,"internally_reserved":0,"allocated":314318848,"data_stored":313466676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2375271987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,14,11,19,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18808832,"num_objects":-2296,"num_object_clones":0,"num_object_copies":-9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"1.f","version":"40'10343","reported_seq":10580,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.304562+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:13.304562+0000","last_peered":"2024-04-26T02:34:13.304562+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:13.304562+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9674752,"num_objects":1181,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10599","reported_seq":11841,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:10.150917+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:10.150917+0000","last_peered":"2024-04-26T02:34:10.150917+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:10.150917+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10097164,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10413","reported_seq":10587,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:11.904145+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:11.904145+0000","last_peered":"2024-04-26T02:34:11.904145+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:11.904145+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9967902,"num_objects":1218,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10412,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10275","reported_seq":14305,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:12.055541+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:12.055541+0000","last_peered":"2024-04-26T02:34:12.055541+0000","last_clean":"2024-04-26T02:34:12.055541+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:12.055541+0000","last_undegraded":"2024-04-26T02:34:12.055541+0000","last_fullsized":"2024-04-26T02:34:12.055541+0000","mapping_epoch":37,"log_start":"28'200","ondisk_log_start":"28'200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:16.003127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9867788,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10379","reported_seq":10614,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.417352+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:13.417352+0000","last_peered":"2024-04-26T02:34:13.417352+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:13.417352+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9974284,"num_objects":1218,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10464","reported_seq":10725,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.056200+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:13.056200+0000","last_peered":"2024-04-26T02:34:13.056200+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:13.056200+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":1136,"num_object_clones":0,"num_object_copies":4544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10548","reported_seq":14805,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:10.097498+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:10.097498+0000","last_peered":"2024-04-26T02:34:10.097498+0000","last_clean":"2024-04-26T02:34:10.097498+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:10.097498+0000","last_undegraded":"2024-04-26T02:34:10.097498+0000","last_fullsized":"2024-04-26T02:34:10.097498+0000","mapping_epoch":33,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9908748,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10828","reported_seq":11087,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:10.079206+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:10.079206+0000","last_peered":"2024-04-26T02:34:10.079206+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:10.079206+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9507340,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10826,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-891}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10541","reported_seq":14636,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:13.058507+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:13.058507+0000","last_peered":"2024-04-26T02:34:13.058507+0000","last_clean":"2024-04-26T02:34:13.058507+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:13.058507+0000","last_undegraded":"2024-04-26T02:34:13.058507+0000","last_fullsized":"2024-04-26T02:34:13.058507+0000","mapping_epoch":39,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:15.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815064,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10538,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10425","reported_seq":10604,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:12.041585+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:12.041585+0000","last_peered":"2024-04-26T02:34:12.041585+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:12.041585+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9327116,"num_objects":1139,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10562","reported_seq":10801,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.320385+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:13.320385+0000","last_peered":"2024-04-26T02:34:13.320385+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:13.320385+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9472786,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10474","reported_seq":10708,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.384368+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:13.384368+0000","last_peered":"2024-04-26T02:34:13.384368+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:13.384368+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9197830,"num_objects":1123,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-775}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10560","reported_seq":10782,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:11.999632+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:11.999632+0000","last_peered":"2024-04-26T02:34:11.999632+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:11.999632+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9984262,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10937","reported_seq":11189,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:13.414583+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:13.414583+0000","last_peered":"2024-04-26T02:34:13.414583+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:13.414583+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9703948,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10642","reported_seq":14857,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:12.128354+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:12.128354+0000","last_peered":"2024-04-26T02:34:12.128354+0000","last_clean":"2024-04-26T02:34:12.128354+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:12.128354+0000","last_undegraded":"2024-04-26T02:34:12.128354+0000","last_fullsized":"2024-04-26T02:34:12.128354+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:13.034324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9933324,"num_objects":1213,"num_object_clones":0,"num_object_copies":4852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10615","reported_seq":10849,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:09.994754+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:09.994754+0000","last_peered":"2024-04-26T02:34:09.994754+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:09.994754+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9679372,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":155417792,"num_objects":18979,"num_object_clones":0,"num_object_copies":75916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168578,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9575,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311898112,"data_stored":311898112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680820,"kb_used_data":75748,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93691020,"statfs":{"total":96636764160,"available":95939604480,"internally_reserved":0,"allocated":77565952,"data_stored":77352909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685276,"kb_used_data":76564,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93686564,"statfs":{"total":96636764160,"available":95935041536,"internally_reserved":0,"allocated":78401536,"data_stored":78188493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573300,"kb_used_data":76588,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93798540,"statfs":{"total":96636764160,"available":96049704960,"internally_reserved":0,"allocated":78426112,"data_stored":78213069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687212,"kb_used_data":78052,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93684628,"statfs":{"total":96636764160,"available":95933059072,"internally_reserved":0,"allocated":79925248,"data_stored":79712205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79376384,"data_stored":79376384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77930496,"data_stored":77930496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77832192,"data_stored":77832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76759040,"data_stored":76759040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:16.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:16.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10343", 'reported_seq': 10580, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.304562+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:13.304562+0000', 'last_peered': '2024-04-26T02:34:13.304562+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:13.304562+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9674752, 'num_objects': 1181, 'num_object_clones': 0, 'num_object_copies': 4724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10343, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -746}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:16.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10599", 'reported_seq': 11841, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:10.150917+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:10.150917+0000', 'last_peered': '2024-04-26T02:34:10.150917+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:10.150917+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10097164, 'num_objects': 1233, 'num_object_clones': 0, 'num_object_copies': 4932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1498, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10597, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 582, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10413", 'reported_seq': 10587, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:11.904145+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:11.904145+0000', 'last_peered': '2024-04-26T02:34:11.904145+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:11.904145+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9967902, 'num_objects': 1218, 'num_object_clones': 0, 'num_object_copies': 4872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1218, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10412, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -766}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10379", 'reported_seq': 10614, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.417352+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:13.417352+0000', 'last_peered': '2024-04-26T02:34:13.417352+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:13.417352+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'300", 'ondisk_log_start': "28'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9974284, 'num_objects': 1218, 'num_object_clones': 0, 'num_object_copies': 4872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10377, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10464", 'reported_seq': 10725, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.056200+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:13.056200+0000', 'last_peered': '2024-04-26T02:34:13.056200+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:13.056200+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9306112, 'num_objects': 1136, 'num_object_clones': 0, 'num_object_copies': 4544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10464, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -814}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10828", 'reported_seq': 11087, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:10.079206+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:10.079206+0000', 'last_peered': '2024-04-26T02:34:10.079206+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:10.079206+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9507340, 'num_objects': 1161, 'num_object_clones': 0, 'num_object_copies': 4644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10826, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -891}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:16.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10425", 'reported_seq': 10604, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:12.041585+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:12.041585+0000', 'last_peered': '2024-04-26T02:34:12.041585+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:12.041585+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9327116, 'num_objects': 1139, 'num_object_clones': 0, 'num_object_copies': 4556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10423, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -856}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10562", 'reported_seq': 10801, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.320385+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:13.320385+0000', 'last_peered': '2024-04-26T02:34:13.320385+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:13.320385+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9472786, 'num_objects': 1157, 'num_object_clones': 0, 'num_object_copies': 4628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10559, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -777}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10474", 'reported_seq': 10708, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.384368+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:13.384368+0000', 'last_peered': '2024-04-26T02:34:13.384368+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:13.384368+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9197830, 'num_objects': 1123, 'num_object_clones': 0, 'num_object_copies': 4492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10473, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -775}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10560", 'reported_seq': 10782, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:11.999632+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:11.999632+0000', 'last_peered': '2024-04-26T02:34:11.999632+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:11.999632+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9984262, 'num_objects': 1219, 'num_object_clones': 0, 'num_object_copies': 4876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10559, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:16.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:16.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'10937", 'reported_seq': 11189, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:13.414583+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:13.414583+0000', 'last_peered': '2024-04-26T02:34:13.414583+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:13.414583+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9703948, 'num_objects': 1185, 'num_object_clones': 0, 'num_object_copies': 4740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10935, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:16.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:16.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10615", 'reported_seq': 10849, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:09.994754+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:09.994754+0000', 'last_peered': '2024-04-26T02:34:09.994754+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:09.994754+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9679372, 'num_objects': 1182, 'num_object_clones': 0, 'num_object_copies': 4728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10613, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:16.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:17.829 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:17.829 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23136/75084 objects degraded (30.813%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":654,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18771,"data_bytes":153713856,"bytes_used":2690023424,"bytes_avail":383857033216,"bytes_total":386547056640,"degraded_objects":23136,"degraded_total":75084,"degraded_ratio":0.30813488892440466,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":172},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-26T02:33:56.711889+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:34:17.900 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:17.900 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:19.712 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:19.713 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:19.848 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-26T02:34:18.721605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147750080,"num_objects":18043,"num_object_clones":0,"num_object_copies":72172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169514,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9633,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2632376,"kb_used_data":291916,"kb_used_omap":14,"kb_used_meta":2340401,"kb_avail":374854984,"statfs":{"total":386547056640,"available":383851503616,"internally_reserved":0,"allocated":298921984,"data_stored":298069812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2396571187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,8,4,2,13,11,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21094400,"num_objects":-2575,"num_object_clones":0,"num_object_copies":-10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003049"},"pg_stats":[{"pgid":"1.f","version":"40'10407","reported_seq":10643,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.568286+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:18.568286+0000","last_peered":"2024-04-26T02:34:18.568286+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:18.568286+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9150464,"num_objects":1117,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10654","reported_seq":12012,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:15.114649+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:15.114649+0000","last_peered":"2024-04-26T02:34:15.114649+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:15.114649+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9646604,"num_objects":1178,"num_object_clones":0,"num_object_copies":4712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10652,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10473","reported_seq":10647,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:16.990457+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:16.990457+0000","last_peered":"2024-04-26T02:34:16.990457+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:16.990457+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9476382,"num_objects":1158,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10472,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10338","reported_seq":14368,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:16.723315+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:16.723315+0000","last_peered":"2024-04-26T02:34:16.723315+0000","last_clean":"2024-04-26T02:34:16.723315+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:16.723315+0000","last_undegraded":"2024-04-26T02:34:16.723315+0000","last_fullsized":"2024-04-26T02:34:16.723315+0000","mapping_epoch":37,"log_start":"28'300","ondisk_log_start":"28'300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:21.031894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9351692,"num_objects":1142,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10336,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10460","reported_seq":10695,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.246392+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:18.246392+0000","last_peered":"2024-04-26T02:34:18.246392+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:18.246392+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9310732,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10519","reported_seq":10780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.518170+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:18.518170+0000","last_peered":"2024-04-26T02:34:18.518170+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:18.518170+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8855552,"num_objects":1081,"num_object_clones":0,"num_object_copies":4324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10611","reported_seq":14868,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:14.926601+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:14.926601+0000","last_peered":"2024-04-26T02:34:14.926601+0000","last_clean":"2024-04-26T02:34:14.926601+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:14.926601+0000","last_undegraded":"2024-04-26T02:34:14.926601+0000","last_fullsized":"2024-04-26T02:34:14.926601+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:15.933691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9392652,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10881","reported_seq":11139,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:14.957709+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:14.957709+0000","last_peered":"2024-04-26T02:34:14.957709+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:14.957709+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9073164,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10612","reported_seq":14707,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:18.376118+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:18.376118+0000","last_peered":"2024-04-26T02:34:18.376118+0000","last_clean":"2024-04-26T02:34:18.376118+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:18.376118+0000","last_undegraded":"2024-04-26T02:34:18.376118+0000","last_fullsized":"2024-04-26T02:34:18.376118+0000","mapping_epoch":39,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:21.766086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9233432,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10472","reported_seq":10651,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:16.863898+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:16.863898+0000","last_peered":"2024-04-26T02:34:16.863898+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:16.863898+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942092,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10609","reported_seq":10848,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.424761+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:18.424761+0000","last_peered":"2024-04-26T02:34:18.424761+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:18.424761+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9087762,"num_objects":1110,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10532","reported_seq":10766,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.568645+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:18.568645+0000","last_peered":"2024-04-26T02:34:18.568645+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:18.568645+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8722694,"num_objects":1065,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-833}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10625","reported_seq":10847,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:17.062226+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:17.062226+0000","last_peered":"2024-04-26T02:34:17.062226+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:17.062226+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9451782,"num_objects":1154,"num_object_clones":0,"num_object_copies":4616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-891}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10997","reported_seq":11249,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.501777+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:18.501777+0000","last_peered":"2024-04-26T02:34:18.501777+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:18.501777+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9212428,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-854}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10699","reported_seq":14914,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:17.174591+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:17.174591+0000","last_peered":"2024-04-26T02:34:17.174591+0000","last_clean":"2024-04-26T02:34:17.174591+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:17.174591+0000","last_undegraded":"2024-04-26T02:34:17.174591+0000","last_fullsized":"2024-04-26T02:34:17.174591+0000","mapping_epoch":35,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:18.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9466380,"num_objects":1156,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10652","reported_seq":10886,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:15.026372+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:15.026372+0000","last_peered":"2024-04-26T02:34:15.026372+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:15.026372+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9376268,"num_objects":1145,"num_object_clones":0,"num_object_copies":4580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":147750080,"num_objects":18043,"num_object_clones":0,"num_object_copies":72172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169514,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9633,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296226816,"data_stored":296226816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681300,"kb_used_data":72076,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93690540,"statfs":{"total":96636764160,"available":95939112960,"internally_reserved":0,"allocated":73805824,"data_stored":73592781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,0,0,0,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685704,"kb_used_data":72832,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93686136,"statfs":{"total":96636764160,"available":95934603264,"internally_reserved":0,"allocated":74579968,"data_stored":74366925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":72748,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":74493952,"data_stored":74280909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687580,"kb_used_data":74260,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93684260,"statfs":{"total":96636764160,"available":95932682240,"internally_reserved":0,"allocated":76042240,"data_stored":75829197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75448320,"data_stored":75448320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74080256,"data_stored":74080256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73887744,"data_stored":73887744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72810496,"data_stored":72810496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:22.850 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:24.545 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:24.551 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:24.650 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-04-26T02:34:22.723132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143137984,"num_objects":17480,"num_object_clones":0,"num_object_copies":69920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170077,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9683,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160604,"ondisk_log_size":160604,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2633512,"kb_used_data":280584,"kb_used_omap":14,"kb_used_meta":2352881,"kb_avail":374853848,"statfs":{"total":386547056640,"available":383850340352,"internally_reserved":0,"allocated":287318016,"data_stored":286465844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2409350707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,4,5,5,12,23,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18292736,"num_objects":-2233,"num_object_clones":0,"num_object_copies":-8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003381"},"pg_stats":[{"pgid":"1.f","version":"40'10407","reported_seq":10643,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.568286+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:18.568286+0000","last_peered":"2024-04-26T02:34:18.568286+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:18.568286+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9150464,"num_objects":1117,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10735","reported_seq":12193,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.362648+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:20.362648+0000","last_peered":"2024-04-26T02:34:20.362648+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:20.362648+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8983052,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10526","reported_seq":10700,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:21.990751+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:21.990751+0000","last_peered":"2024-04-26T02:34:21.990751+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:21.990751+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9042206,"num_objects":1105,"num_object_clones":0,"num_object_copies":4420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1105,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10525,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-879}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10407","reported_seq":14437,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:22.230710+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:22.230710+0000","last_peered":"2024-04-26T02:34:22.230710+0000","last_clean":"2024-04-26T02:34:22.230710+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:22.230710+0000","last_undegraded":"2024-04-26T02:34:22.230710+0000","last_fullsized":"2024-04-26T02:34:22.230710+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:26.076596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786444,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10460","reported_seq":10695,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.246392+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:18.246392+0000","last_peered":"2024-04-26T02:34:18.246392+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:18.246392+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9310732,"num_objects":1137,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10519","reported_seq":10780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.518170+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:18.518170+0000","last_peered":"2024-04-26T02:34:18.518170+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:18.518170+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8855552,"num_objects":1081,"num_object_clones":0,"num_object_copies":4324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10672","reported_seq":14929,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:20.342748+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:20.342748+0000","last_peered":"2024-04-26T02:34:20.342748+0000","last_clean":"2024-04-26T02:34:20.342748+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:20.342748+0000","last_undegraded":"2024-04-26T02:34:20.342748+0000","last_fullsized":"2024-04-26T02:34:20.342748+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:21.871302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8892940,"num_objects":1086,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10670,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10943","reported_seq":11202,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.277991+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:20.277991+0000","last_peered":"2024-04-26T02:34:20.277991+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:20.277991+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8565260,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10612","reported_seq":14707,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:18.376118+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:18.376118+0000","last_peered":"2024-04-26T02:34:18.376118+0000","last_clean":"2024-04-26T02:34:18.376118+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:18.376118+0000","last_undegraded":"2024-04-26T02:34:18.376118+0000","last_fullsized":"2024-04-26T02:34:18.376118+0000","mapping_epoch":39,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:21.766086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9233432,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10522","reported_seq":10698,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:22.102662+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:22.102662+0000","last_peered":"2024-04-26T02:34:22.102662+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:22.102662+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8532492,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10520,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10609","reported_seq":10848,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.424761+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:18.424761+0000","last_peered":"2024-04-26T02:34:18.424761+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:18.424761+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9087762,"num_objects":1110,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10532","reported_seq":10766,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.568645+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:18.568645+0000","last_peered":"2024-04-26T02:34:18.568645+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:18.568645+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8722694,"num_objects":1065,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-833}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10684","reported_seq":10906,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:21.898664+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:21.898664+0000","last_peered":"2024-04-26T02:34:21.898664+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:21.898664+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8968454,"num_objects":1095,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'10997","reported_seq":11249,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:18.501777+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:18.501777+0000","last_peered":"2024-04-26T02:34:18.501777+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:18.501777+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9212428,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-854}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10767","reported_seq":14982,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:22.245033+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:22.245033+0000","last_peered":"2024-04-26T02:34:22.245033+0000","last_clean":"2024-04-26T02:34:22.245033+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:22.245033+0000","last_undegraded":"2024-04-26T02:34:22.245033+0000","last_fullsized":"2024-04-26T02:34:22.245033+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:24.023958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8909324,"num_objects":1088,"num_object_clones":0,"num_object_copies":4352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10765,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10712","reported_seq":10946,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.378077+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:20.378077+0000","last_peered":"2024-04-26T02:34:20.378077+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:20.378077+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8884748,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":143137984,"num_objects":17480,"num_object_clones":0,"num_object_copies":69920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170077,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9683,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":284082176,"data_stored":284082176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160604,"ondisk_log_size":160604,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681300,"kb_used_data":72076,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93690540,"statfs":{"total":96636764160,"available":95939112960,"internally_reserved":0,"allocated":73805824,"data_stored":73592781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,0,0,0,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686148,"kb_used_data":69116,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93685692,"statfs":{"total":96636764160,"available":95934148608,"internally_reserved":0,"allocated":70774784,"data_stored":70561741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578124,"kb_used_data":68932,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93793716,"statfs":{"total":96636764160,"available":96044765184,"internally_reserved":0,"allocated":70586368,"data_stored":70373325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,8,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687940,"kb_used_data":70460,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93683900,"statfs":{"total":96636764160,"available":95932313600,"internally_reserved":0,"allocated":72151040,"data_stored":71937997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71376896,"data_stored":71376896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69988352,"data_stored":69988352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69906432,"data_stored":69906432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72810496,"data_stored":72810496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:24.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:26.394 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:26.508 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-26T02:34:24.726543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139713728,"num_objects":17062,"num_object_clones":0,"num_object_copies":68248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170495,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9683,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2633792,"kb_used_data":276704,"kb_used_omap":14,"kb_used_meta":2357041,"kb_avail":374853568,"statfs":{"total":386547056640,"available":383850053632,"internally_reserved":0,"allocated":283344896,"data_stored":282492724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2413610547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,5,6,14,21,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19513344,"num_objects":-2382,"num_object_clones":0,"num_object_copies":-9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006319"},"pg_stats":[{"pgid":"1.f","version":"40'10463","reported_seq":10700,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.622906+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:23.622906+0000","last_peered":"2024-04-26T02:34:23.622906+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:23.622906+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10463,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10735","reported_seq":12193,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.362648+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:20.362648+0000","last_peered":"2024-04-26T02:34:20.362648+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:20.362648+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8983052,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10526","reported_seq":10700,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:21.990751+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:21.990751+0000","last_peered":"2024-04-26T02:34:21.990751+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:21.990751+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9042206,"num_objects":1105,"num_object_clones":0,"num_object_copies":4420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1105,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10525,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-879}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10407","reported_seq":14437,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:22.230710+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:22.230710+0000","last_peered":"2024-04-26T02:34:22.230710+0000","last_clean":"2024-04-26T02:34:22.230710+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:22.230710+0000","last_undegraded":"2024-04-26T02:34:22.230710+0000","last_fullsized":"2024-04-26T02:34:22.230710+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:26.076596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786444,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10524","reported_seq":10758,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.526498+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:23.526498+0000","last_peered":"2024-04-26T02:34:23.526498+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:23.526498+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786444,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10522,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10582","reported_seq":10843,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.506386+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:23.506386+0000","last_peered":"2024-04-26T02:34:23.506386+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:23.506386+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8339456,"num_objects":1018,"num_object_clones":0,"num_object_copies":4072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10672","reported_seq":14929,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:20.342748+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:20.342748+0000","last_peered":"2024-04-26T02:34:20.342748+0000","last_clean":"2024-04-26T02:34:20.342748+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:20.342748+0000","last_undegraded":"2024-04-26T02:34:20.342748+0000","last_fullsized":"2024-04-26T02:34:20.342748+0000","mapping_epoch":33,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:21.871302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8892940,"num_objects":1086,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10670,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10943","reported_seq":11202,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.277991+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:20.277991+0000","last_peered":"2024-04-26T02:34:20.277991+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:20.277991+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8565260,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1006}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10668","reported_seq":14763,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:23.602943+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:23.602943+0000","last_peered":"2024-04-26T02:34:23.602943+0000","last_clean":"2024-04-26T02:34:23.602943+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:23.602943+0000","last_undegraded":"2024-04-26T02:34:23.602943+0000","last_fullsized":"2024-04-26T02:34:23.602943+0000","mapping_epoch":39,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:26.823176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8774680,"num_objects":1072,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10522","reported_seq":10698,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:22.102662+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:22.102662+0000","last_peered":"2024-04-26T02:34:22.102662+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:22.102662+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8532492,"num_objects":1042,"num_object_clones":0,"num_object_copies":4168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10520,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10669","reported_seq":10908,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.385940+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:23.385940+0000","last_peered":"2024-04-26T02:34:23.385940+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:23.385940+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8596242,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10584","reported_seq":10817,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.661797+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:23.661797+0000","last_peered":"2024-04-26T02:34:23.661797+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:23.661797+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8296710,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10583,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-885}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10684","reported_seq":10906,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:21.898664+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:21.898664+0000","last_peered":"2024-04-26T02:34:21.898664+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:21.898664+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8968454,"num_objects":1095,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11064","reported_seq":11316,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.526007+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:23.526007+0000","last_peered":"2024-04-26T02:34:23.526007+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:23.526007+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8663564,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11062,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-921}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10767","reported_seq":14982,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:22.245033+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:22.245033+0000","last_peered":"2024-04-26T02:34:22.245033+0000","last_clean":"2024-04-26T02:34:22.245033+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:22.245033+0000","last_undegraded":"2024-04-26T02:34:22.245033+0000","last_fullsized":"2024-04-26T02:34:22.245033+0000","mapping_epoch":35,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:24.023958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8909324,"num_objects":1088,"num_object_clones":0,"num_object_copies":4352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10765,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10712","reported_seq":10946,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:20.378077+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:20.378077+0000","last_peered":"2024-04-26T02:34:20.378077+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:20.378077+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8884748,"num_objects":1085,"num_object_clones":0,"num_object_copies":4340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139713728,"num_objects":17062,"num_object_clones":0,"num_object_copies":68248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170495,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9683,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280047616,"data_stored":280047616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681580,"kb_used_data":68196,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93690260,"statfs":{"total":96636764160,"available":95938826240,"internally_reserved":0,"allocated":69832704,"data_stored":69619661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686148,"kb_used_data":69116,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93685692,"statfs":{"total":96636764160,"available":95934148608,"internally_reserved":0,"allocated":70774784,"data_stored":70561741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578124,"kb_used_data":68932,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93793716,"statfs":{"total":96636764160,"available":96044765184,"internally_reserved":0,"allocated":70586368,"data_stored":70373325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,8,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687940,"kb_used_data":70460,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93683900,"statfs":{"total":96636764160,"available":95932313600,"internally_reserved":0,"allocated":72151040,"data_stored":71937997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71376896,"data_stored":71376896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69988352,"data_stored":69988352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69906432,"data_stored":69906432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68775936,"data_stored":68775936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10463", 'reported_seq': 10700, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.622906+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:23.622906+0000', 'last_peered': '2024-04-26T02:34:23.622906+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:23.622906+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'400", 'ondisk_log_start': "28'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8691712, 'num_objects': 1061, 'num_object_clones': 0, 'num_object_copies': 4244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10463, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10735", 'reported_seq': 12193, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:20.362648+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:20.362648+0000', 'last_peered': '2024-04-26T02:34:20.362648+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:20.362648+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8983052, 'num_objects': 1097, 'num_object_clones': 0, 'num_object_copies': 4388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1390, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10733, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 690, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10526", 'reported_seq': 10700, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:21.990751+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:21.990751+0000', 'last_peered': '2024-04-26T02:34:21.990751+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:21.990751+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9042206, 'num_objects': 1105, 'num_object_clones': 0, 'num_object_copies': 4420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1105, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10525, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -879}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:26.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10524", 'reported_seq': 10758, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.526498+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:23.526498+0000', 'last_peered': '2024-04-26T02:34:23.526498+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:23.526498+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8786444, 'num_objects': 1073, 'num_object_clones': 0, 'num_object_copies': 4292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10522, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -856}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10582", 'reported_seq': 10843, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.506386+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:23.506386+0000', 'last_peered': '2024-04-26T02:34:23.506386+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:23.506386+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8339456, 'num_objects': 1018, 'num_object_clones': 0, 'num_object_copies': 4072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10582, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'10943", 'reported_seq': 11202, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:20.277991+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:20.277991+0000', 'last_peered': '2024-04-26T02:34:20.277991+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:20.277991+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8565260, 'num_objects': 1046, 'num_object_clones': 0, 'num_object_copies': 4184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10522", 'reported_seq': 10698, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:22.102662+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:22.102662+0000', 'last_peered': '2024-04-26T02:34:22.102662+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:22.102662+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8532492, 'num_objects': 1042, 'num_object_clones': 0, 'num_object_copies': 4168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10520, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -953}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:26.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10669", 'reported_seq': 10908, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.385940+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:23.385940+0000', 'last_peered': '2024-04-26T02:34:23.385940+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:23.385940+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8596242, 'num_objects': 1050, 'num_object_clones': 0, 'num_object_copies': 4200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10666, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -884}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10584", 'reported_seq': 10817, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.661797+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:23.661797+0000', 'last_peered': '2024-04-26T02:34:23.661797+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:23.661797+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8296710, 'num_objects': 1013, 'num_object_clones': 0, 'num_object_copies': 4052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10583, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10684", 'reported_seq': 10906, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:21.898664+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:21.898664+0000', 'last_peered': '2024-04-26T02:34:21.898664+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:21.898664+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8968454, 'num_objects': 1095, 'num_object_clones': 0, 'num_object_copies': 4380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10683, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11064", 'reported_seq': 11316, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:23.526007+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:23.526007+0000', 'last_peered': '2024-04-26T02:34:23.526007+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:23.526007+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8663564, 'num_objects': 1058, 'num_object_clones': 0, 'num_object_copies': 4232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11062, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -921}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:26.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10712", 'reported_seq': 10946, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:20.378077+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:20.378077+0000', 'last_peered': '2024-04-26T02:34:20.378077+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:20.378077+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8884748, 'num_objects': 1085, 'num_object_clones': 0, 'num_object_copies': 4340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10710, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:26.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:28.446 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:28.446 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23036/67324 objects degraded (34.217%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":664,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16831,"data_bytes":137821376,"bytes_used":2698739712,"bytes_avail":383848316928,"bytes_total":386547056640,"degraded_objects":23036,"degraded_total":67324,"degraded_ratio":0.34216624086507041,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-04-26T02:33:56.711889+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:34:28.559 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:28.559 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:30.037 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:30.137 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-04-26T02:34:28.727531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135216320,"num_objects":16513,"num_object_clones":0,"num_object_copies":66052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171044,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9733,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160971,"ondisk_log_size":160971,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625124,"kb_used_data":265220,"kb_used_omap":14,"kb_used_meta":2359857,"kb_avail":374862236,"statfs":{"total":386547056640,"available":383858929664,"internally_reserved":0,"allocated":271585280,"data_stored":270733108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2416494131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,9,3,11,12,11,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18497536,"num_objects":-2258,"num_object_clones":0,"num_object_copies":-9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006331"},"pg_stats":[{"pgid":"1.f","version":"40'10463","reported_seq":10700,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.622906+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:23.622906+0000","last_peered":"2024-04-26T02:34:23.622906+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:23.622906+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10463,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10803","reported_seq":12359,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:25.396268+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:25.396268+0000","last_peered":"2024-04-26T02:34:25.396268+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:25.396268+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8425996,"num_objects":1029,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10583","reported_seq":10757,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:27.025637+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:27.025637+0000","last_peered":"2024-04-26T02:34:27.025637+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:27.025637+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8575262,"num_objects":1048,"num_object_clones":0,"num_object_copies":4192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10582,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10472","reported_seq":14502,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:27.524824+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:27.524824+0000","last_peered":"2024-04-26T02:34:27.524824+0000","last_clean":"2024-04-26T02:34:27.524824+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:27.524824+0000","last_undegraded":"2024-04-26T02:34:27.524824+0000","last_fullsized":"2024-04-26T02:34:27.524824+0000","mapping_epoch":37,"log_start":"28'400","ondisk_log_start":"28'400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:32.083328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253964,"num_objects":1008,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10524","reported_seq":10758,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.526498+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:23.526498+0000","last_peered":"2024-04-26T02:34:23.526498+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:23.526498+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786444,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10522,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10582","reported_seq":10843,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.506386+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:23.506386+0000","last_peered":"2024-04-26T02:34:23.506386+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:23.506386+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8339456,"num_objects":1018,"num_object_clones":0,"num_object_copies":4072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10726","reported_seq":14983,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:25.272038+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:25.272038+0000","last_peered":"2024-04-26T02:34:25.272038+0000","last_clean":"2024-04-26T02:34:25.272038+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:25.272038+0000","last_undegraded":"2024-04-26T02:34:25.272038+0000","last_fullsized":"2024-04-26T02:34:25.272038+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:26.889873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8450572,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'10998","reported_seq":11257,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:25.216006+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:25.216006+0000","last_peered":"2024-04-26T02:34:25.216006+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:25.216006+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8114700,"num_objects":991,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10996,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10668","reported_seq":14763,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:23.602943+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:23.602943+0000","last_peered":"2024-04-26T02:34:23.602943+0000","last_clean":"2024-04-26T02:34:23.602943+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:23.602943+0000","last_undegraded":"2024-04-26T02:34:23.602943+0000","last_fullsized":"2024-04-26T02:34:23.602943+0000","mapping_epoch":39,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:26.823176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8774680,"num_objects":1072,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10591","reported_seq":10770,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:27.227996+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:27.227996+0000","last_peered":"2024-04-26T02:34:27.227996+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:27.227996+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7967244,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10669","reported_seq":10908,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.385940+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:23.385940+0000","last_peered":"2024-04-26T02:34:23.385940+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:23.385940+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8596242,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10584","reported_seq":10817,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.661797+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:23.661797+0000","last_peered":"2024-04-26T02:34:23.661797+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:23.661797+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8296710,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10583,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-885}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10747","reported_seq":10969,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:27.031017+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:27.031017+0000","last_peered":"2024-04-26T02:34:27.031017+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:27.031017+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8452358,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10746,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11064","reported_seq":11316,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:23.526007+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:23.526007+0000","last_peered":"2024-04-26T02:34:23.526007+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:23.526007+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8663564,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11062,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-921}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10831","reported_seq":15045,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:27.587419+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:27.587419+0000","last_peered":"2024-04-26T02:34:27.587419+0000","last_clean":"2024-04-26T02:34:27.587419+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:27.587419+0000","last_undegraded":"2024-04-26T02:34:27.587419+0000","last_fullsized":"2024-04-26T02:34:27.587419+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:30.060828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8385036,"num_objects":1024,"num_object_clones":0,"num_object_copies":4096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10766","reported_seq":11000,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:25.234619+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:25.234619+0000","last_peered":"2024-04-26T02:34:25.234619+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:25.234619+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8442380,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135216320,"num_objects":16513,"num_object_clones":0,"num_object_copies":66052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171044,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9733,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268177408,"data_stored":268177408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160971,"ondisk_log_size":160971,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681580,"kb_used_data":68196,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93690260,"statfs":{"total":96636764160,"available":95938826240,"internally_reserved":0,"allocated":69832704,"data_stored":69619661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692060,"kb_used_data":65300,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93679780,"statfs":{"total":96636764160,"available":95928094720,"internally_reserved":0,"allocated":66867200,"data_stored":66654157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561848,"kb_used_data":65136,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93809992,"statfs":{"total":96636764160,"available":96061431808,"internally_reserved":0,"allocated":66699264,"data_stored":66486221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,4,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689636,"kb_used_data":66588,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93682204,"statfs":{"total":96636764160,"available":95930576896,"internally_reserved":0,"allocated":68186112,"data_stored":67973069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":637988559},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67407872,"data_stored":67407872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65945600,"data_stored":65945600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66048000,"data_stored":66048000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68775936,"data_stored":68775936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:33.139 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:34.689 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:34.691 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:34.827 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-04-26T02:34:32.728563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127294656,"num_objects":15546,"num_object_clones":0,"num_object_copies":62184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15546,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172011,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9789,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160738,"ondisk_log_size":160738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2644720,"kb_used_data":250096,"kb_used_omap":14,"kb_used_meta":2394545,"kb_avail":374842640,"statfs":{"total":386547056640,"available":383838863360,"internally_reserved":0,"allocated":256098304,"data_stored":255246132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2452014643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,11,4,14,12,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18292736,"num_objects":-2233,"num_object_clones":0,"num_object_copies":-8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005974"},"pg_stats":[{"pgid":"1.f","version":"40'10533","reported_seq":10767,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.535399+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:28.535399+0000","last_peered":"2024-04-26T02:34:28.535399+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:28.535399+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118272,"num_objects":991,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10533,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10859","reported_seq":12529,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.508317+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:30.508317+0000","last_peered":"2024-04-26T02:34:30.508317+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:30.508317+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7967244,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10647","reported_seq":10821,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.018427+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:32.018427+0000","last_peered":"2024-04-26T02:34:32.018427+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:32.018427+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050974,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10646,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10525","reported_seq":14555,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:32.351103+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:32.351103+0000","last_peered":"2024-04-26T02:34:32.351103+0000","last_clean":"2024-04-26T02:34:32.351103+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:32.351103+0000","last_undegraded":"2024-04-26T02:34:32.351103+0000","last_fullsized":"2024-04-26T02:34:32.351103+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:37.091121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7819788,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10586","reported_seq":10822,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.628700+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:28.628700+0000","last_peered":"2024-04-26T02:34:28.628700+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:28.628700+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8278540,"num_objects":1011,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10649","reported_seq":10910,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.581474+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:28.581474+0000","last_peered":"2024-04-26T02:34:28.581474+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:28.581474+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7790592,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10649,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10783","reported_seq":15040,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:30.434068+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:30.434068+0000","last_peered":"2024-04-26T02:34:30.434068+0000","last_clean":"2024-04-26T02:34:30.434068+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:30.434068+0000","last_undegraded":"2024-04-26T02:34:30.434068+0000","last_fullsized":"2024-04-26T02:34:30.434068+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:32.864541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7983628,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11055","reported_seq":11314,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.421005+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:30.421005+0000","last_peered":"2024-04-26T02:34:30.421005+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:30.421005+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647756,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10728","reported_seq":14823,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:28.653126+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:28.653126+0000","last_peered":"2024-04-26T02:34:28.653126+0000","last_clean":"2024-04-26T02:34:28.653126+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:28.653126+0000","last_undegraded":"2024-04-26T02:34:28.653126+0000","last_fullsized":"2024-04-26T02:34:28.653126+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:31.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8283160,"num_objects":1012,"num_object_clones":0,"num_object_copies":4048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10647","reported_seq":10826,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.075496+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:32.075496+0000","last_peered":"2024-04-26T02:34:32.075496+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:32.075496+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7508492,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10733","reported_seq":10972,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.578719+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:28.578719+0000","last_peered":"2024-04-26T02:34:28.578719+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:28.578719+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8071954,"num_objects":986,"num_object_clones":0,"num_object_copies":3944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10633","reported_seq":10867,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.628882+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:28.628882+0000","last_peered":"2024-04-26T02:34:28.628882+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:28.628882+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7895302,"num_objects":964,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10805","reported_seq":11027,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.043961+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:32.043961+0000","last_peered":"2024-04-26T02:34:32.043961+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:32.043961+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7977222,"num_objects":974,"num_object_clones":0,"num_object_copies":3896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11128","reported_seq":11380,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:28.627450+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:28.627450+0000","last_peered":"2024-04-26T02:34:28.627450+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:28.627450+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8139276,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-985}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10898","reported_seq":15113,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:32.662588+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:32.662588+0000","last_peered":"2024-04-26T02:34:32.662588+0000","last_clean":"2024-04-26T02:34:32.662588+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:32.662588+0000","last_undegraded":"2024-04-26T02:34:32.662588+0000","last_fullsized":"2024-04-26T02:34:32.662588+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:36.122694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836172,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10829","reported_seq":11063,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.433032+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:30.433032+0000","last_peered":"2024-04-26T02:34:30.433032+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:30.433032+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7926284,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":127294656,"num_objects":15546,"num_object_clones":0,"num_object_copies":62184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15546,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172011,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9789,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":252432384,"data_stored":252432384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160738,"ondisk_log_size":160738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699992,"kb_used_data":64392,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93671848,"statfs":{"total":96636764160,"available":95919972352,"internally_reserved":0,"allocated":65937408,"data_stored":65724365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":650833616},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692460,"kb_used_data":61524,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93679380,"statfs":{"total":96636764160,"available":95927685120,"internally_reserved":0,"allocated":63000576,"data_stored":62787533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562212,"kb_used_data":61336,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93809628,"statfs":{"total":96636764160,"available":96061059072,"internally_reserved":0,"allocated":62808064,"data_stored":62595021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,1,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690056,"kb_used_data":62844,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93681784,"statfs":{"total":96636764160,"available":95930146816,"internally_reserved":0,"allocated":64352256,"data_stored":64139213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63475712,"data_stored":63475712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61992960,"data_stored":61992960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62132224,"data_stored":62132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64831488,"data_stored":64831488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:34.828 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:36.213 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:36.214 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:36.343 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-26T02:34:34.729070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123813056,"num_objects":15121,"num_object_clones":0,"num_object_copies":60484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172436,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9789,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161063,"ondisk_log_size":161063,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2631796,"kb_used_data":245500,"kb_used_omap":14,"kb_used_meta":2386225,"kb_avail":374855564,"statfs":{"total":386547056640,"available":383852097536,"internally_reserved":0,"allocated":251392000,"data_stored":250539828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2443494963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,17,3,5,9,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19324928,"num_objects":-2359,"num_object_clones":0,"num_object_copies":-9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005938"},"pg_stats":[{"pgid":"1.f","version":"40'10595","reported_seq":10832,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.763278+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:33.763278+0000","last_peered":"2024-04-26T02:34:33.763278+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:33.763278+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7610368,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10859","reported_seq":12529,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.508317+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:30.508317+0000","last_peered":"2024-04-26T02:34:30.508317+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:30.508317+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7967244,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10647","reported_seq":10821,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.018427+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:32.018427+0000","last_peered":"2024-04-26T02:34:32.018427+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:32.018427+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050974,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10646,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1000}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10525","reported_seq":14555,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:32.351103+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:32.351103+0000","last_peered":"2024-04-26T02:34:32.351103+0000","last_clean":"2024-04-26T02:34:32.351103+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:32.351103+0000","last_undegraded":"2024-04-26T02:34:32.351103+0000","last_fullsized":"2024-04-26T02:34:32.351103+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:37.091121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7819788,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10660","reported_seq":10896,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.680929+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:33.680929+0000","last_peered":"2024-04-26T02:34:33.680929+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:33.680929+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672332,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10700","reported_seq":10961,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.607618+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:33.607618+0000","last_peered":"2024-04-26T02:34:33.607618+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:33.607618+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7372800,"num_objects":900,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10783","reported_seq":15040,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:30.434068+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:30.434068+0000","last_peered":"2024-04-26T02:34:30.434068+0000","last_clean":"2024-04-26T02:34:30.434068+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:30.434068+0000","last_undegraded":"2024-04-26T02:34:30.434068+0000","last_fullsized":"2024-04-26T02:34:30.434068+0000","mapping_epoch":33,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:32.864541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7983628,"num_objects":975,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11055","reported_seq":11314,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.421005+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:30.421005+0000","last_peered":"2024-04-26T02:34:30.421005+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:30.421005+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647756,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10784","reported_seq":14879,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:33.572447+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:33.572447+0000","last_peered":"2024-04-26T02:34:33.572447+0000","last_clean":"2024-04-26T02:34:33.572447+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:33.572447+0000","last_undegraded":"2024-04-26T02:34:33.572447+0000","last_fullsized":"2024-04-26T02:34:33.572447+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:37.795461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7824408,"num_objects":956,"num_object_clones":0,"num_object_copies":3824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10647","reported_seq":10826,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.075496+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:32.075496+0000","last_peered":"2024-04-26T02:34:32.075496+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:32.075496+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7508492,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10790","reported_seq":11029,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.738452+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:33.738452+0000","last_peered":"2024-04-26T02:34:33.738452+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:33.738452+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7605010,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10691","reported_seq":10924,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.759151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:33.759151+0000","last_peered":"2024-04-26T02:34:33.759151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:33.759151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7420166,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10690,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10805","reported_seq":11027,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:32.043961+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:32.043961+0000","last_peered":"2024-04-26T02:34:32.043961+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:32.043961+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7977222,"num_objects":974,"num_object_clones":0,"num_object_copies":3896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1071}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11195","reported_seq":11447,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.681468+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:33.681468+0000","last_peered":"2024-04-26T02:34:33.681468+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:33.681468+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7590412,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10898","reported_seq":15113,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:32.662588+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:32.662588+0000","last_peered":"2024-04-26T02:34:32.662588+0000","last_clean":"2024-04-26T02:34:32.662588+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:32.662588+0000","last_undegraded":"2024-04-26T02:34:32.662588+0000","last_fullsized":"2024-04-26T02:34:32.662588+0000","mapping_epoch":35,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:36.122694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836172,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10829","reported_seq":11063,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:30.433032+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:30.433032+0000","last_peered":"2024-04-26T02:34:30.433032+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:30.433032+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7926284,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":123813056,"num_objects":15121,"num_object_clones":0,"num_object_copies":60484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172436,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9789,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":248451072,"data_stored":248451072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161063,"ondisk_log_size":161063,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687068,"kb_used_data":59796,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93684772,"statfs":{"total":96636764160,"available":95933206528,"internally_reserved":0,"allocated":61231104,"data_stored":61018061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692460,"kb_used_data":61524,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93679380,"statfs":{"total":96636764160,"available":95927685120,"internally_reserved":0,"allocated":63000576,"data_stored":62787533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562212,"kb_used_data":61336,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93809628,"statfs":{"total":96636764160,"available":96061059072,"internally_reserved":0,"allocated":62808064,"data_stored":62595021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,1,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690056,"kb_used_data":62844,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93681784,"statfs":{"total":96636764160,"available":95930146816,"internally_reserved":0,"allocated":64352256,"data_stored":64139213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642248399},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63475712,"data_stored":63475712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61992960,"data_stored":61992960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62132224,"data_stored":62132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60850176,"data_stored":60850176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:36.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10595", 'reported_seq': 10832, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.763278+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:33.763278+0000', 'last_peered': '2024-04-26T02:34:33.763278+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:33.763278+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'500", 'ondisk_log_start': "28'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7610368, 'num_objects': 929, 'num_object_clones': 0, 'num_object_copies': 3716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10595, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10859", 'reported_seq': 12529, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:30.508317+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:30.508317+0000', 'last_peered': '2024-04-26T02:34:30.508317+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:30.508317+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7967244, 'num_objects': 973, 'num_object_clones': 0, 'num_object_copies': 3892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10857, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 796, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -311}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10647", 'reported_seq': 10821, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:32.018427+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:32.018427+0000', 'last_peered': '2024-04-26T02:34:32.018427+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:32.018427+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8050974, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10646, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1000}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10660", 'reported_seq': 10896, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.680929+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:33.680929+0000', 'last_peered': '2024-04-26T02:34:33.680929+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:33.680929+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7672332, 'num_objects': 937, 'num_object_clones': 0, 'num_object_copies': 3748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10658, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:36.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10700", 'reported_seq': 10961, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.607618+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:33.607618+0000', 'last_peered': '2024-04-26T02:34:33.607618+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:33.607618+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7372800, 'num_objects': 900, 'num_object_clones': 0, 'num_object_copies': 3600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10700, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1050}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11055", 'reported_seq': 11314, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:30.421005+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:30.421005+0000', 'last_peered': '2024-04-26T02:34:30.421005+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:30.421005+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7647756, 'num_objects': 934, 'num_object_clones': 0, 'num_object_copies': 3736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11053, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10647", 'reported_seq': 10826, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:32.075496+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:32.075496+0000', 'last_peered': '2024-04-26T02:34:32.075496+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:32.075496+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7508492, 'num_objects': 917, 'num_object_clones': 0, 'num_object_copies': 3668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10645, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1078}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10790", 'reported_seq': 11029, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.738452+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:33.738452+0000', 'last_peered': '2024-04-26T02:34:33.738452+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:33.738452+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7605010, 'num_objects': 929, 'num_object_clones': 0, 'num_object_copies': 3716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10787, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1005}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10691", 'reported_seq': 10924, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.759151+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:33.759151+0000', 'last_peered': '2024-04-26T02:34:33.759151+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:33.759151+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'600", 'ondisk_log_start': "28'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7420166, 'num_objects': 906, 'num_object_clones': 0, 'num_object_copies': 3624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10690, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:36.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10805", 'reported_seq': 11027, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:32.043961+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:32.043961+0000', 'last_peered': '2024-04-26T02:34:32.043961+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:32.043961+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7977222, 'num_objects': 974, 'num_object_clones': 0, 'num_object_copies': 3896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10804, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:36.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:36.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11195", 'reported_seq': 11447, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:33.681468+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:33.681468+0000', 'last_peered': '2024-04-26T02:34:33.681468+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:33.681468+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7590412, 'num_objects': 927, 'num_object_clones': 0, 'num_object_copies': 3708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:36.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:36.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10829", 'reported_seq': 11063, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:30.433032+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:30.433032+0000', 'last_peered': '2024-04-26T02:34:30.433032+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:30.433032+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7926284, 'num_objects': 968, 'num_object_clones': 0, 'num_object_copies': 3872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10827, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:36.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:38.166 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:38.167 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22911/59540 objects degraded (38.480%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":674,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14885,"data_bytes":121879744,"bytes_used":2695294976,"bytes_avail":383851761664,"bytes_total":386547056640,"degraded_objects":22911,"degraded_total":59540,"degraded_ratio":0.38480013436345312,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:34:38.229 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:38.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:40.074 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:40.075 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:40.145 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-04-26T02:34:38.730004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119356608,"num_objects":14577,"num_object_clones":0,"num_object_copies":58308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172980,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9858,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2650840,"kb_used_data":234028,"kb_used_omap":14,"kb_used_meta":2416753,"kb_avail":374836520,"statfs":{"total":386547056640,"available":383832596480,"internally_reserved":0,"allocated":239644672,"data_stored":238792500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2474755635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,14,6,5,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18464768,"num_objects":-2254,"num_object_clones":0,"num_object_copies":-9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.f","version":"40'10595","reported_seq":10832,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.763278+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:33.763278+0000","last_peered":"2024-04-26T02:34:33.763278+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:33.763278+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7610368,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10921","reported_seq":12728,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:35.489742+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:35.489742+0000","last_peered":"2024-04-26T02:34:35.489742+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:35.489742+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7459340,"num_objects":911,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-304}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10702","reported_seq":10876,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:37.178252+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:37.178252+0000","last_peered":"2024-04-26T02:34:37.178252+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:37.178252+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7600414,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10701,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10587","reported_seq":14617,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:37.628341+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:37.628341+0000","last_peered":"2024-04-26T02:34:37.628341+0000","last_clean":"2024-04-26T02:34:37.628341+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:37.628341+0000","last_undegraded":"2024-04-26T02:34:37.628341+0000","last_fullsized":"2024-04-26T02:34:37.628341+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:42.097392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7311884,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10660","reported_seq":10896,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.680929+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:33.680929+0000","last_peered":"2024-04-26T02:34:33.680929+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:33.680929+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7672332,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10700","reported_seq":10961,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.607618+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:33.607618+0000","last_peered":"2024-04-26T02:34:33.607618+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:33.607618+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7372800,"num_objects":900,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10846","reported_seq":15103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:35.411562+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:35.411562+0000","last_peered":"2024-04-26T02:34:35.411562+0000","last_clean":"2024-04-26T02:34:35.411562+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:35.411562+0000","last_undegraded":"2024-04-26T02:34:35.411562+0000","last_fullsized":"2024-04-26T02:34:35.411562+0000","mapping_epoch":33,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:37.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7467532,"num_objects":912,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11115","reported_seq":11374,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:35.186722+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:35.186722+0000","last_peered":"2024-04-26T02:34:35.186722+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:35.186722+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7156236,"num_objects":874,"num_object_clones":0,"num_object_copies":3496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11113,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10784","reported_seq":14879,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:33.572447+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:33.572447+0000","last_peered":"2024-04-26T02:34:33.572447+0000","last_clean":"2024-04-26T02:34:33.572447+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:33.572447+0000","last_undegraded":"2024-04-26T02:34:33.572447+0000","last_fullsized":"2024-04-26T02:34:33.572447+0000","mapping_epoch":39,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:37.795461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7824408,"num_objects":956,"num_object_clones":0,"num_object_copies":3824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10712","reported_seq":10891,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:37.275037+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:37.275037+0000","last_peered":"2024-04-26T02:34:37.275037+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:37.275037+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6976012,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10790","reported_seq":11029,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.738452+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:33.738452+0000","last_peered":"2024-04-26T02:34:33.738452+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:33.738452+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7605010,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10691","reported_seq":10924,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.759151+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:33.759151+0000","last_peered":"2024-04-26T02:34:33.759151+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:33.759151+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7420166,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10690,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10865","reported_seq":11087,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:37.178917+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:37.178917+0000","last_peered":"2024-04-26T02:34:37.178917+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:37.178917+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7485702,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10864,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11195","reported_seq":11447,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:33.681468+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:33.681468+0000","last_peered":"2024-04-26T02:34:33.681468+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:33.681468+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7590412,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10964","reported_seq":15179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:37.598272+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:37.598272+0000","last_peered":"2024-04-26T02:34:37.598272+0000","last_clean":"2024-04-26T02:34:37.598272+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:37.598272+0000","last_undegraded":"2024-04-26T02:34:37.598272+0000","last_fullsized":"2024-04-26T02:34:37.598272+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:41.125048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7295500,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10962,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10880","reported_seq":11114,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:35.360949+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:35.360949+0000","last_peered":"2024-04-26T02:34:35.360949+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:35.360949+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7508492,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":119356608,"num_objects":14577,"num_object_clones":0,"num_object_copies":58308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172980,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9858,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":236646400,"data_stored":236646400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687068,"kb_used_data":59796,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93684772,"statfs":{"total":96636764160,"available":95933206528,"internally_reserved":0,"allocated":61231104,"data_stored":61018061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710816,"kb_used_data":57688,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93661024,"statfs":{"total":96636764160,"available":95908888576,"internally_reserved":0,"allocated":59072512,"data_stored":58859469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562572,"kb_used_data":57528,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93809268,"statfs":{"total":96636764160,"available":96060690432,"internally_reserved":0,"allocated":58908672,"data_stored":58695629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690384,"kb_used_data":59016,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93681456,"statfs":{"total":96636764160,"available":95929810944,"internally_reserved":0,"allocated":60432384,"data_stored":60219341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646508239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,2,3,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":59551744,"data_stored":59551744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58060800,"data_stored":58060800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58183680,"data_stored":58183680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60850176,"data_stored":60850176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:43.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:44.645 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:44.646 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:44.728 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-04-26T02:34:42.730992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112377024,"num_objects":13725,"num_object_clones":0,"num_object_copies":54900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173832,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9899,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2671976,"kb_used_data":223216,"kb_used_omap":14,"kb_used_meta":2448689,"kb_avail":374815384,"statfs":{"total":386547056640,"available":383810953216,"internally_reserved":0,"allocated":228573184,"data_stored":227721012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2507458099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,9,7,4,12,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17358848,"num_objects":-2119,"num_object_clones":0,"num_object_copies":-8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"1.f","version":"40'10655","reported_seq":10892,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.742437+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:38.742437+0000","last_peered":"2024-04-26T02:34:38.742437+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:38.742437+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7118848,"num_objects":869,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10655,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1058}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10981","reported_seq":12870,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.632223+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:40.632223+0000","last_peered":"2024-04-26T02:34:40.632223+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:40.632223+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6967820,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10979,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10773","reported_seq":10947,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.282336+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:42.282336+0000","last_peered":"2024-04-26T02:34:42.282336+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:42.282336+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7018782,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10772,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10587","reported_seq":14617,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:37.628341+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:37.628341+0000","last_peered":"2024-04-26T02:34:37.628341+0000","last_clean":"2024-04-26T02:34:37.628341+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:37.628341+0000","last_undegraded":"2024-04-26T02:34:37.628341+0000","last_fullsized":"2024-04-26T02:34:37.628341+0000","mapping_epoch":37,"log_start":"28'500","ondisk_log_start":"28'500","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:42.097392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7311884,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10710","reported_seq":10946,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.511174+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:38.511174+0000","last_peered":"2024-04-26T02:34:38.511174+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:38.511174+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7262732,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10708,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10761","reported_seq":11022,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.639638+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:38.639638+0000","last_peered":"2024-04-26T02:34:38.639638+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:38.639638+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6873088,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10918","reported_seq":15175,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:40.477863+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:40.477863+0000","last_peered":"2024-04-26T02:34:40.477863+0000","last_clean":"2024-04-26T02:34:40.477863+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:40.477863+0000","last_undegraded":"2024-04-26T02:34:40.477863+0000","last_fullsized":"2024-04-26T02:34:40.477863+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:44.027096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6877708,"num_objects":840,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11180","reported_seq":11439,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.638701+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:40.638701+0000","last_peered":"2024-04-26T02:34:40.638701+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:40.638701+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6623756,"num_objects":809,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10843","reported_seq":14938,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:38.833274+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:38.833274+0000","last_peered":"2024-04-26T02:34:38.833274+0000","last_clean":"2024-04-26T02:34:38.833274+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:38.833274+0000","last_undegraded":"2024-04-26T02:34:38.833274+0000","last_fullsized":"2024-04-26T02:34:38.833274+0000","mapping_epoch":39,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:42.995086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7341080,"num_objects":897,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10774","reported_seq":10953,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.130462+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:42.130462+0000","last_peered":"2024-04-26T02:34:42.130462+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:42.130462+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6468108,"num_objects":790,"num_object_clones":0,"num_object_copies":3160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10852","reported_seq":11091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.640681+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:38.640681+0000","last_peered":"2024-04-26T02:34:38.640681+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:38.640681+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7097106,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10751","reported_seq":10985,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.774159+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:38.774159+0000","last_peered":"2024-04-26T02:34:38.774159+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:38.774159+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6928646,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10750,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10929","reported_seq":11151,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.209483+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:42.209483+0000","last_peered":"2024-04-26T02:34:42.209483+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:42.209483+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6961414,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11238","reported_seq":11490,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:38.784932+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:38.784932+0000","last_peered":"2024-04-26T02:34:38.784932+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:38.784932+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7238156,"num_objects":884,"num_object_clones":0,"num_object_copies":3536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1095}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'10964","reported_seq":15179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:37.598272+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:37.598272+0000","last_peered":"2024-04-26T02:34:37.598272+0000","last_clean":"2024-04-26T02:34:37.598272+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:37.598272+0000","last_undegraded":"2024-04-26T02:34:37.598272+0000","last_fullsized":"2024-04-26T02:34:37.598272+0000","mapping_epoch":35,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:41.125048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7295500,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10962,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10943","reported_seq":11177,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.506083+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:40.506083+0000","last_peered":"2024-04-26T02:34:40.506083+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:40.506083+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6992396,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":112377024,"num_objects":13725,"num_object_clones":0,"num_object_copies":54900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173832,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9899,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":224870400,"data_stored":224870400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706144,"kb_used_data":56652,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93665696,"statfs":{"total":96636764160,"available":95913672704,"internally_reserved":0,"allocated":58011648,"data_stored":57798605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":53904,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":55197696,"data_stored":54984653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562572,"kb_used_data":57528,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93809268,"statfs":{"total":96636764160,"available":96060690432,"internally_reserved":0,"allocated":58908672,"data_stored":58695629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708708,"kb_used_data":55132,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93663132,"statfs":{"total":96636764160,"available":95911047168,"internally_reserved":0,"allocated":56455168,"data_stored":56242125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,2,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55595008,"data_stored":55595008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58060800,"data_stored":58060800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54235136,"data_stored":54235136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56979456,"data_stored":56979456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:44.729 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:46.415 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:46.415 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:46.568 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-04-26T02:34:44.731473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108027072,"num_objects":13194,"num_object_clones":0,"num_object_copies":52776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13194,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174363,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9899,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2676856,"kb_used_data":215636,"kb_used_omap":14,"kb_used_meta":2461169,"kb_avail":374810504,"statfs":{"total":386547056640,"available":383805956096,"internally_reserved":0,"allocated":220811264,"data_stored":219959092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2520237619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,1,9,18,11,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19267584,"num_objects":-2352,"num_object_clones":0,"num_object_copies":-9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002910"},"pg_stats":[{"pgid":"1.f","version":"40'10717","reported_seq":10952,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.717900+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:43.717900+0000","last_peered":"2024-04-26T02:34:43.717900+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:43.717900+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'10981","reported_seq":12870,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.632223+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:40.632223+0000","last_peered":"2024-04-26T02:34:40.632223+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:40.632223+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6967820,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10979,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10773","reported_seq":10947,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.282336+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:42.282336+0000","last_peered":"2024-04-26T02:34:42.282336+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:42.282336+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7018782,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10772,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10646","reported_seq":14676,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:42.719436+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:42.719436+0000","last_peered":"2024-04-26T02:34:42.719436+0000","last_clean":"2024-04-26T02:34:42.719436+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:42.719436+0000","last_undegraded":"2024-04-26T02:34:42.719436+0000","last_fullsized":"2024-04-26T02:34:42.719436+0000","mapping_epoch":37,"log_start":"28'600","ondisk_log_start":"28'600","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:47.121965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6828556,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10768","reported_seq":11004,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.869673+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:43.869673+0000","last_peered":"2024-04-26T02:34:43.869673+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:43.869673+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6787596,"num_objects":829,"num_object_clones":0,"num_object_copies":3316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10817","reported_seq":11078,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.750586+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:43.750586+0000","last_peered":"2024-04-26T02:34:43.750586+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:43.750586+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6414336,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10918","reported_seq":15175,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:40.477863+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:40.477863+0000","last_peered":"2024-04-26T02:34:40.477863+0000","last_clean":"2024-04-26T02:34:40.477863+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:40.477863+0000","last_undegraded":"2024-04-26T02:34:40.477863+0000","last_fullsized":"2024-04-26T02:34:40.477863+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:44.027096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6877708,"num_objects":840,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11180","reported_seq":11439,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.638701+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:40.638701+0000","last_peered":"2024-04-26T02:34:40.638701+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:40.638701+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6623756,"num_objects":809,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10899","reported_seq":14994,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:43.905504+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:43.905504+0000","last_peered":"2024-04-26T02:34:43.905504+0000","last_clean":"2024-04-26T02:34:43.905504+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:43.905504+0000","last_undegraded":"2024-04-26T02:34:43.905504+0000","last_fullsized":"2024-04-26T02:34:43.905504+0000","mapping_epoch":39,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6882328,"num_objects":841,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10774","reported_seq":10953,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.130462+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:42.130462+0000","last_peered":"2024-04-26T02:34:42.130462+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:42.130462+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6468108,"num_objects":790,"num_object_clones":0,"num_object_copies":3160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10909","reported_seq":11148,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.788867+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:43.788867+0000","last_peered":"2024-04-26T02:34:43.788867+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:43.788867+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6630162,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10816","reported_seq":11050,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.901005+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:43.901005+0000","last_peered":"2024-04-26T02:34:43.901005+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:43.901005+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6396166,"num_objects":781,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10815,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1117}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10929","reported_seq":11151,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:42.209483+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:42.209483+0000","last_peered":"2024-04-26T02:34:42.209483+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:42.209483+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6961414,"num_objects":850,"num_object_clones":0,"num_object_copies":3400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11299","reported_seq":11551,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.645305+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:43.645305+0000","last_peered":"2024-04-26T02:34:43.645305+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:43.645305+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6738444,"num_objects":823,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11021","reported_seq":15235,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:42.669465+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:42.669465+0000","last_peered":"2024-04-26T02:34:42.669465+0000","last_clean":"2024-04-26T02:34:42.669465+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:42.669465+0000","last_undegraded":"2024-04-26T02:34:42.669465+0000","last_fullsized":"2024-04-26T02:34:42.669465+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:47.121942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6828556,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'10943","reported_seq":11177,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:40.506083+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:40.506083+0000","last_peered":"2024-04-26T02:34:40.506083+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:40.506083+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6992396,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108027072,"num_objects":13194,"num_object_clones":0,"num_object_copies":52776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13194,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174363,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9899,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216920064,"data_stored":216920064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706520,"kb_used_data":52876,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93665320,"statfs":{"total":96636764160,"available":95913287680,"internally_reserved":0,"allocated":54145024,"data_stored":53931981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,4,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":53904,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":55197696,"data_stored":54984653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567076,"kb_used_data":53724,"kb_used_omap":4,"kb_used_meta":513339,"kb_avail":93804764,"statfs":{"total":96636764160,"available":96056078336,"internally_reserved":0,"allocated":55013376,"data_stored":54800333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525659855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,11,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708708,"kb_used_data":55132,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93663132,"statfs":{"total":96636764160,"available":95911047168,"internally_reserved":0,"allocated":56455168,"data_stored":56242125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,2,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55595008,"data_stored":55595008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54071296,"data_stored":54071296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54235136,"data_stored":54235136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53018624,"data_stored":53018624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:46.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:46.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10717", 'reported_seq': 10952, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.717900+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:43.717900+0000', 'last_peered': '2024-04-26T02:34:43.717900+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:43.717900+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6610944, 'num_objects': 807, 'num_object_clones': 0, 'num_object_copies': 3228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10717, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1120}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'10981", 'reported_seq': 12870, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:40.632223+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:40.632223+0000', 'last_peered': '2024-04-26T02:34:40.632223+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:40.632223+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6967820, 'num_objects': 851, 'num_object_clones': 0, 'num_object_copies': 3404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1174, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10979, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 906, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10773", 'reported_seq': 10947, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:42.282336+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:42.282336+0000', 'last_peered': '2024-04-26T02:34:42.282336+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:42.282336+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7018782, 'num_objects': 858, 'num_object_clones': 0, 'num_object_copies': 3432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 858, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10772, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1126}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10768", 'reported_seq': 11004, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.869673+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:43.869673+0000', 'last_peered': '2024-04-26T02:34:43.869673+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:43.869673+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6787596, 'num_objects': 829, 'num_object_clones': 0, 'num_object_copies': 3316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10766, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10817", 'reported_seq': 11078, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.750586+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:43.750586+0000', 'last_peered': '2024-04-26T02:34:43.750586+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:43.750586+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6414336, 'num_objects': 783, 'num_object_clones': 0, 'num_object_copies': 3132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10817, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11180", 'reported_seq': 11439, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:40.638701+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:40.638701+0000', 'last_peered': '2024-04-26T02:34:40.638701+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:40.638701+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6623756, 'num_objects': 809, 'num_object_clones': 0, 'num_object_copies': 3236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11178, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10774", 'reported_seq': 10953, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:42.130462+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:42.130462+0000', 'last_peered': '2024-04-26T02:34:42.130462+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:42.130462+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'700", 'ondisk_log_start': "28'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6468108, 'num_objects': 790, 'num_object_clones': 0, 'num_object_copies': 3160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10772, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1205}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:46.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'10909", 'reported_seq': 11148, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.788867+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:43.788867+0000', 'last_peered': '2024-04-26T02:34:43.788867+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:43.788867+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6630162, 'num_objects': 810, 'num_object_clones': 0, 'num_object_copies': 3240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10906, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10816", 'reported_seq': 11050, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.901005+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:43.901005+0000', 'last_peered': '2024-04-26T02:34:43.901005+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:43.901005+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6396166, 'num_objects': 781, 'num_object_clones': 0, 'num_object_copies': 3124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10815, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'10929", 'reported_seq': 11151, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:42.209483+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:42.209483+0000', 'last_peered': '2024-04-26T02:34:42.209483+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:42.209483+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6961414, 'num_objects': 850, 'num_object_clones': 0, 'num_object_copies': 3400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10928, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11299", 'reported_seq': 11551, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:43.645305+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:43.645305+0000', 'last_peered': '2024-04-26T02:34:43.645305+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:43.645305+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6738444, 'num_objects': 823, 'num_object_clones': 0, 'num_object_copies': 3292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11297, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:46.572 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'10943", 'reported_seq': 11177, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:40.506083+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:40.506083+0000', 'last_peered': '2024-04-26T02:34:40.506083+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:40.506083+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6992396, 'num_objects': 854, 'num_object_clones': 0, 'num_object_copies': 3416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:46.573 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:47.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:47.727 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22807/51832 objects degraded (44.002%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":684,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12958,"data_bytes":106093760,"bytes_used":2728636416,"bytes_avail":383818420224,"bytes_total":386547056640,"degraded_objects":22807,"degraded_total":51832,"degraded_ratio":0.44001774965272417,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:34:47.834 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:47.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:49.686 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:49.686 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:49.801 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-26T02:34:48.734666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103652544,"num_objects":12660,"num_object_clones":0,"num_object_copies":50640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174897,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9962,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2648812,"kb_used_data":204216,"kb_used_omap":14,"kb_used_meta":2444529,"kb_avail":374838548,"statfs":{"total":386547056640,"available":383834673152,"internally_reserved":0,"allocated":209117184,"data_stored":208265012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2503198259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,10,14,11,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18227200,"num_objects":-2225,"num_object_clones":0,"num_object_copies":-8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005106"},"pg_stats":[{"pgid":"1.f","version":"40'10717","reported_seq":10952,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.717900+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:43.717900+0000","last_peered":"2024-04-26T02:34:43.717900+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:43.717900+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6610944,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11038","reported_seq":13054,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:45.732835+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:45.732835+0000","last_peered":"2024-04-26T02:34:45.732835+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:45.732835+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6500876,"num_objects":794,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11036,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":969,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10832","reported_seq":11006,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:47.204344+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:47.204344+0000","last_peered":"2024-04-26T02:34:47.204344+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:47.204344+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6535454,"num_objects":799,"num_object_clones":0,"num_object_copies":3196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10831,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10709","reported_seq":14739,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:47.918077+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:47.918077+0000","last_peered":"2024-04-26T02:34:47.918077+0000","last_clean":"2024-04-26T02:34:47.918077+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:47.918077+0000","last_undegraded":"2024-04-26T02:34:47.918077+0000","last_fullsized":"2024-04-26T02:34:47.918077+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6312460,"num_objects":771,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10768","reported_seq":11004,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.869673+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:43.869673+0000","last_peered":"2024-04-26T02:34:43.869673+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:43.869673+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6787596,"num_objects":829,"num_object_clones":0,"num_object_copies":3316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10817","reported_seq":11078,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.750586+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:43.750586+0000","last_peered":"2024-04-26T02:34:43.750586+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:43.750586+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6414336,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'10975","reported_seq":15233,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:45.641341+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:45.641341+0000","last_peered":"2024-04-26T02:34:45.641341+0000","last_clean":"2024-04-26T02:34:45.641341+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:45.641341+0000","last_undegraded":"2024-04-26T02:34:45.641341+0000","last_fullsized":"2024-04-26T02:34:45.641341+0000","mapping_epoch":33,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:49.086567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6410764,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11237","reported_seq":11494,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:45.570777+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:45.570777+0000","last_peered":"2024-04-26T02:34:45.570777+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:45.570777+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6156812,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10899","reported_seq":14994,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:43.905504+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:43.905504+0000","last_peered":"2024-04-26T02:34:43.905504+0000","last_clean":"2024-04-26T02:34:43.905504+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:43.905504+0000","last_undegraded":"2024-04-26T02:34:43.905504+0000","last_fullsized":"2024-04-26T02:34:43.905504+0000","mapping_epoch":39,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6882328,"num_objects":841,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10833","reported_seq":11012,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:47.363975+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:47.363975+0000","last_peered":"2024-04-26T02:34:47.363975+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:47.363975+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5984780,"num_objects":731,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10909","reported_seq":11148,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.788867+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:43.788867+0000","last_peered":"2024-04-26T02:34:43.788867+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:43.788867+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6630162,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10816","reported_seq":11050,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.901005+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:43.901005+0000","last_peered":"2024-04-26T02:34:43.901005+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:43.901005+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6396166,"num_objects":781,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10815,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1117}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'10988","reported_seq":11210,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:47.108858+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:47.108858+0000","last_peered":"2024-04-26T02:34:47.108858+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:47.108858+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6478086,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10987,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11299","reported_seq":11551,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:43.645305+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:43.645305+0000","last_peered":"2024-04-26T02:34:43.645305+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:43.645305+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6738444,"num_objects":823,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11079","reported_seq":15294,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:47.927078+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:47.927078+0000","last_peered":"2024-04-26T02:34:47.927078+0000","last_clean":"2024-04-26T02:34:47.927078+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:47.927078+0000","last_undegraded":"2024-04-26T02:34:47.927078+0000","last_fullsized":"2024-04-26T02:34:47.927078+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353420,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11077,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11008","reported_seq":11242,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:45.573160+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:45.573160+0000","last_peered":"2024-04-26T02:34:45.573160+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:45.573160+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6459916,"num_objects":789,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11006,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":103652544,"num_objects":12660,"num_object_clones":0,"num_object_copies":50640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174897,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9962,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205180928,"data_stored":205180928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706520,"kb_used_data":52876,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93665320,"statfs":{"total":96636764160,"available":95913287680,"internally_reserved":0,"allocated":54145024,"data_stored":53931981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,4,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694912,"kb_used_data":50104,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93676928,"statfs":{"total":96636764160,"available":95925174272,"internally_reserved":0,"allocated":51306496,"data_stored":51093453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550844,"kb_used_data":49972,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93820996,"statfs":{"total":96636764160,"available":96072699904,"internally_reserved":0,"allocated":51171328,"data_stored":50958285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696536,"kb_used_data":51264,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93675304,"statfs":{"total":96636764160,"available":95923511296,"internally_reserved":0,"allocated":52494336,"data_stored":52281293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51621888,"data_stored":51621888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50196480,"data_stored":50196480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50343936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53018624,"data_stored":53018624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:52.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:54.742 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:54.848 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-04-26T02:34:52.740666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97057984,"num_objects":11855,"num_object_clones":0,"num_object_copies":47420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175702,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10037,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161029,"ondisk_log_size":161029,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2633388,"kb_used_data":192944,"kb_used_omap":14,"kb_used_meta":2440369,"kb_avail":374853972,"statfs":{"total":386547056640,"available":383850467328,"internally_reserved":0,"allocated":197574656,"data_stored":196722484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2498938419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,11,17,6,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16932864,"num_objects":-2067,"num_object_clones":0,"num_object_copies":-8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010173"},"pg_stats":[{"pgid":"1.f","version":"40'10775","reported_seq":11011,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.952325+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:48.952325+0000","last_peered":"2024-04-26T02:34:48.952325+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:48.952325+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135808,"num_objects":749,"num_object_clones":0,"num_object_copies":2996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11099","reported_seq":13265,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.737237+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:50.737237+0000","last_peered":"2024-04-26T02:34:50.737237+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:50.737237+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6001164,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11097,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10888","reported_seq":11062,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.078757+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:52.078757+0000","last_peered":"2024-04-26T02:34:52.078757+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:52.078757+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6076702,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10887,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10709","reported_seq":14739,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:47.918077+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:47.918077+0000","last_peered":"2024-04-26T02:34:47.918077+0000","last_clean":"2024-04-26T02:34:47.918077+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:47.918077+0000","last_undegraded":"2024-04-26T02:34:47.918077+0000","last_fullsized":"2024-04-26T02:34:47.918077+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6312460,"num_objects":771,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10830","reported_seq":11066,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.944782+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:48.944782+0000","last_peered":"2024-04-26T02:34:48.944782+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:48.944782+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6279692,"num_objects":767,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10872","reported_seq":11133,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.995290+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:48.995290+0000","last_peered":"2024-04-26T02:34:48.995290+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:48.995290+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":728,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10872,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11031","reported_seq":15289,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:50.422809+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:50.422809+0000","last_peered":"2024-04-26T02:34:50.422809+0000","last_clean":"2024-04-26T02:34:50.422809+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:50.422809+0000","last_undegraded":"2024-04-26T02:34:50.422809+0000","last_fullsized":"2024-04-26T02:34:50.422809+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.191853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5952012,"num_objects":727,"num_object_clones":0,"num_object_copies":2908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11029,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11284","reported_seq":11544,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.794242+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:50.794242+0000","last_peered":"2024-04-26T02:34:50.794242+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:50.794242+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5771788,"num_objects":705,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'10963","reported_seq":15059,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:48.908420+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:48.908420+0000","last_peered":"2024-04-26T02:34:48.908420+0000","last_clean":"2024-04-26T02:34:48.908420+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:48.908420+0000","last_undegraded":"2024-04-26T02:34:48.908420+0000","last_fullsized":"2024-04-26T02:34:48.908420+0000","mapping_epoch":39,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6358040,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10886","reported_seq":11065,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.243835+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:52.243835+0000","last_peered":"2024-04-26T02:34:52.243835+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:52.243835+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5550604,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'10957","reported_seq":11196,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.809423+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:48.809423+0000","last_peered":"2024-04-26T02:34:48.809423+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:48.809423+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6236946,"num_objects":762,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10954,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10881","reported_seq":11115,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.820267+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:48.820267+0000","last_peered":"2024-04-26T02:34:48.820267+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:48.820267+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5863686,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1182}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11047","reported_seq":11269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.407671+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:52.407671+0000","last_peered":"2024-04-26T02:34:52.407671+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:52.407671+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5994758,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1313}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11355","reported_seq":11607,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:48.744230+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:48.744230+0000","last_peered":"2024-04-26T02:34:48.744230+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:48.744230+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6279692,"num_objects":767,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11079","reported_seq":15294,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:47.927078+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:47.927078+0000","last_peered":"2024-04-26T02:34:47.927078+0000","last_clean":"2024-04-26T02:34:47.927078+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:47.927078+0000","last_undegraded":"2024-04-26T02:34:47.927078+0000","last_fullsized":"2024-04-26T02:34:47.927078+0000","mapping_epoch":35,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353420,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11077,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11073","reported_seq":11307,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.508858+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:50.508858+0000","last_peered":"2024-04-26T02:34:50.508858+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:50.508858+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5927436,"num_objects":724,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":97057984,"num_objects":11855,"num_object_clones":0,"num_object_copies":47420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175702,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10037,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193470464,"data_stored":193470464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161029,"ondisk_log_size":161029,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690312,"kb_used_data":49148,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93681528,"statfs":{"total":96636764160,"available":95929884672,"internally_reserved":0,"allocated":50327552,"data_stored":50114509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,6,2,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695348,"kb_used_data":46372,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93676492,"statfs":{"total":96636764160,"available":95924727808,"internally_reserved":0,"allocated":47484928,"data_stored":47271885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550844,"kb_used_data":49972,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93820996,"statfs":{"total":96636764160,"available":96072699904,"internally_reserved":0,"allocated":51171328,"data_stored":50958285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696884,"kb_used_data":47452,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93674956,"statfs":{"total":96636764160,"available":95923154944,"internally_reserved":0,"allocated":48590848,"data_stored":48377805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47714304,"data_stored":47714304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50196480,"data_stored":50196480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46387200,"data_stored":46387200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49172480,"data_stored":49172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:54.849 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:56.003 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:56.003 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:34:56.118 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-26T02:34:54.741147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92372160,"num_objects":11283,"num_object_clones":0,"num_object_copies":45132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176274,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10037,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160901,"ondisk_log_size":160901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2650416,"kb_used_data":183584,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374836944,"statfs":{"total":386547056640,"available":383833030656,"internally_reserved":0,"allocated":187990016,"data_stored":187137844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2525939251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,10,9,18,8,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20004864,"num_objects":-2442,"num_object_clones":0,"num_object_copies":-9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010154"},"pg_stats":[{"pgid":"1.f","version":"40'10827","reported_seq":11064,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:54.050110+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:54.050110+0000","last_peered":"2024-04-26T02:34:54.050110+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:54.050110+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5709824,"num_objects":697,"num_object_clones":0,"num_object_copies":2788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11099","reported_seq":13265,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.737237+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:50.737237+0000","last_peered":"2024-04-26T02:34:50.737237+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:50.737237+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6001164,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11097,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10888","reported_seq":11062,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.078757+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:52.078757+0000","last_peered":"2024-04-26T02:34:52.078757+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:52.078757+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6076702,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10887,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10770","reported_seq":14801,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:53.094636+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:53.094636+0000","last_peered":"2024-04-26T02:34:53.094636+0000","last_clean":"2024-04-26T02:34:53.094636+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:53.094636+0000","last_undegraded":"2024-04-26T02:34:53.094636+0000","last_fullsized":"2024-04-26T02:34:53.094636+0000","mapping_epoch":37,"log_start":"28'700","ondisk_log_start":"28'700","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:58.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5812748,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10768,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10894","reported_seq":11130,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.875167+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:53.875167+0000","last_peered":"2024-04-26T02:34:53.875167+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:53.875167+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5755404,"num_objects":703,"num_object_clones":0,"num_object_copies":2812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10892,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10931","reported_seq":11192,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.977903+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:53.977903+0000","last_peered":"2024-04-26T02:34:53.977903+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:53.977903+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11031","reported_seq":15289,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:50.422809+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:50.422809+0000","last_peered":"2024-04-26T02:34:50.422809+0000","last_clean":"2024-04-26T02:34:50.422809+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:50.422809+0000","last_undegraded":"2024-04-26T02:34:50.422809+0000","last_fullsized":"2024-04-26T02:34:50.422809+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.191853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5952012,"num_objects":727,"num_object_clones":0,"num_object_copies":2908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11029,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11284","reported_seq":11544,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.794242+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:50.794242+0000","last_peered":"2024-04-26T02:34:50.794242+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:50.794242+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5771788,"num_objects":705,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11034","reported_seq":15130,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:53.984785+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:53.984785+0000","last_peered":"2024-04-26T02:34:53.984785+0000","last_clean":"2024-04-26T02:34:53.984785+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:53.984785+0000","last_undegraded":"2024-04-26T02:34:53.984785+0000","last_fullsized":"2024-04-26T02:34:53.984785+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.040487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5776408,"num_objects":706,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11031,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10886","reported_seq":11065,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.243835+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:52.243835+0000","last_peered":"2024-04-26T02:34:52.243835+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:52.243835+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5550604,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11020","reported_seq":11260,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.750891+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:53.750891+0000","last_peered":"2024-04-26T02:34:53.750891+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:53.750891+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5720850,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10940","reported_seq":11174,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.750155+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:53.750155+0000","last_peered":"2024-04-26T02:34:53.750155+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:53.750155+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5380358,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1241}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11047","reported_seq":11269,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:52.407671+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:52.407671+0000","last_peered":"2024-04-26T02:34:52.407671+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:52.407671+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5994758,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1313}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11419","reported_seq":11671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.941703+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:53.941703+0000","last_peered":"2024-04-26T02:34:53.941703+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:53.941703+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5755404,"num_objects":703,"num_object_clones":0,"num_object_copies":2812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1276}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11158","reported_seq":15373,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:53.110171+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:53.110171+0000","last_peered":"2024-04-26T02:34:53.110171+0000","last_clean":"2024-04-26T02:34:53.110171+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:53.110171+0000","last_undegraded":"2024-04-26T02:34:53.110171+0000","last_fullsized":"2024-04-26T02:34:53.110171+0000","mapping_epoch":35,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:58.090920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5706252,"num_objects":697,"num_object_clones":0,"num_object_copies":2788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11073","reported_seq":11307,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:50.508858+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:50.508858+0000","last_peered":"2024-04-26T02:34:50.508858+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:50.508858+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5927436,"num_objects":724,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1263}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":92372160,"num_objects":11283,"num_object_clones":0,"num_object_copies":45132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176274,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10037,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185176064,"data_stored":185176064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160901,"ondisk_log_size":160901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689760,"kb_used_data":44432,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93682080,"statfs":{"total":96636764160,"available":95930449920,"internally_reserved":0,"allocated":45498368,"data_stored":45285325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,10,1,6,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695348,"kb_used_data":46372,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93676492,"statfs":{"total":96636764160,"available":95924727808,"internally_reserved":0,"allocated":47484928,"data_stored":47271885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849018999,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568424,"kb_used_data":45328,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93803416,"statfs":{"total":96636764160,"available":96054697984,"internally_reserved":0,"allocated":46415872,"data_stored":46202829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,6,2,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696884,"kb_used_data":47452,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93674956,"statfs":{"total":96636764160,"available":95923154944,"internally_reserved":0,"allocated":48590848,"data_stored":48377805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47714304,"data_stored":47714304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45973504,"data_stored":45973504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46387200,"data_stored":46387200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45101056,"data_stored":45101056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:34:56.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:34:56.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10827", 'reported_seq': 11064, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:54.050110+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:34:54.050110+0000', 'last_peered': '2024-04-26T02:34:54.050110+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:34:54.050110+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5709824, 'num_objects': 697, 'num_object_clones': 0, 'num_object_copies': 2788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10827, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1230}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11099", 'reported_seq': 13265, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:50.737237+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:34:50.737237+0000', 'last_peered': '2024-04-26T02:34:50.737237+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:34:50.737237+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6001164, 'num_objects': 733, 'num_object_clones': 0, 'num_object_copies': 2932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11097, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1044, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'10888", 'reported_seq': 11062, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:52.078757+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:34:52.078757+0000', 'last_peered': '2024-04-26T02:34:52.078757+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:34:52.078757+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6076702, 'num_objects': 743, 'num_object_clones': 0, 'num_object_copies': 2972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 743, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10887, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1241}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'10894", 'reported_seq': 11130, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:53.875167+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:34:53.875167+0000', 'last_peered': '2024-04-26T02:34:53.875167+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:34:53.875167+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5755404, 'num_objects': 703, 'num_object_clones': 0, 'num_object_copies': 2812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10892, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1226}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'10931", 'reported_seq': 11192, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:53.977903+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:34:53.977903+0000', 'last_peered': '2024-04-26T02:34:53.977903+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:34:53.977903+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5480448, 'num_objects': 669, 'num_object_clones': 0, 'num_object_copies': 2676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10931, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11284", 'reported_seq': 11544, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:50.794242+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:34:50.794242+0000', 'last_peered': '2024-04-26T02:34:50.794242+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:34:50.794242+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5771788, 'num_objects': 705, 'num_object_clones': 0, 'num_object_copies': 2820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11282, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:34:56.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'10886", 'reported_seq': 11065, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:52.243835+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:34:52.243835+0000', 'last_peered': '2024-04-26T02:34:52.243835+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:34:52.243835+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'800", 'ondisk_log_start': "28'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5550604, 'num_objects': 678, 'num_object_clones': 0, 'num_object_copies': 2712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10884, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1317}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11020", 'reported_seq': 11260, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:53.750891+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:34:53.750891+0000', 'last_peered': '2024-04-26T02:34:53.750891+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:34:53.750891+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5720850, 'num_objects': 699, 'num_object_clones': 0, 'num_object_copies': 2796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11017, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1235}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'10940", 'reported_seq': 11174, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:53.750155+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:34:53.750155+0000', 'last_peered': '2024-04-26T02:34:53.750155+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:34:53.750155+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5380358, 'num_objects': 657, 'num_object_clones': 0, 'num_object_copies': 2628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10939, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11047", 'reported_seq': 11269, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:52.407671+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:34:52.407671+0000', 'last_peered': '2024-04-26T02:34:52.407671+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:34:52.407671+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5994758, 'num_objects': 732, 'num_object_clones': 0, 'num_object_copies': 2928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11046, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11419", 'reported_seq': 11671, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:53.941703+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:34:53.941703+0000', 'last_peered': '2024-04-26T02:34:53.941703+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:34:53.941703+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5755404, 'num_objects': 703, 'num_object_clones': 0, 'num_object_copies': 2812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11417, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1276}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:34:56.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11073", 'reported_seq': 11307, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:34:50.508858+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:34:50.508858+0000', 'last_peered': '2024-04-26T02:34:50.508858+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:34:50.508858+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5927436, 'num_objects': 724, 'num_object_clones': 0, 'num_object_copies': 2896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11071, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:34:56.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:34:57.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:57.975 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22677/44132 objects degraded (51.384%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":694,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11033,"data_bytes":90324160,"bytes_used":2714374144,"bytes_avail":383832682496,"bytes_total":386547056640,"degraded_objects":22677,"degraded_total":44132,"degraded_ratio":0.51384482914891694,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:34:58.070 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:34:58.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:34:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:34:59.846 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:00.004 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-04-26T02:34:58.742145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87932096,"num_objects":10741,"num_object_clones":0,"num_object_copies":42964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176816,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10092,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2655628,"kb_used_data":172196,"kb_used_omap":14,"kb_used_meta":2483377,"kb_avail":374831732,"statfs":{"total":386547056640,"available":383827693568,"internally_reserved":0,"allocated":176328704,"data_stored":175476532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2542978611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,15,5,25,13,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18161664,"num_objects":-2217,"num_object_clones":0,"num_object_copies":-8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007967"},"pg_stats":[{"pgid":"1.f","version":"40'10827","reported_seq":11064,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:54.050110+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:54.050110+0000","last_peered":"2024-04-26T02:34:54.050110+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:54.050110+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5709824,"num_objects":697,"num_object_clones":0,"num_object_copies":2788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11163","reported_seq":13439,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:55.876921+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:34:55.876921+0000","last_peered":"2024-04-26T02:34:55.876921+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:34:55.876921+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5476876,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1099,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'10960","reported_seq":11134,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:57.381245+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:34:57.381245+0000","last_peered":"2024-04-26T02:34:57.381245+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:34:57.381245+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5486878,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10959,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1313}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10823","reported_seq":14854,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:58.194280+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:58.194280+0000","last_peered":"2024-04-26T02:34:58.194280+0000","last_clean":"2024-04-26T02:34:58.194280+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:58.194280+0000","last_undegraded":"2024-04-26T02:34:58.194280+0000","last_fullsized":"2024-04-26T02:34:58.194280+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378572,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10821,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10894","reported_seq":11130,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.875167+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:53.875167+0000","last_peered":"2024-04-26T02:34:53.875167+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:53.875167+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5755404,"num_objects":703,"num_object_clones":0,"num_object_copies":2812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10892,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10931","reported_seq":11192,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.977903+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:53.977903+0000","last_peered":"2024-04-26T02:34:53.977903+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:53.977903+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11094","reported_seq":15352,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:55.615127+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:34:55.615127+0000","last_peered":"2024-04-26T02:34:55.615127+0000","last_clean":"2024-04-26T02:34:55.615127+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:34:55.615127+0000","last_undegraded":"2024-04-26T02:34:55.615127+0000","last_fullsized":"2024-04-26T02:34:55.615127+0000","mapping_epoch":33,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:00.119445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5435916,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11092,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11347","reported_seq":11607,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:55.698065+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:34:55.698065+0000","last_peered":"2024-04-26T02:34:55.698065+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:34:55.698065+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5255692,"num_objects":642,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11034","reported_seq":15130,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:53.984785+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:53.984785+0000","last_peered":"2024-04-26T02:34:53.984785+0000","last_clean":"2024-04-26T02:34:53.984785+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:53.984785+0000","last_undegraded":"2024-04-26T02:34:53.984785+0000","last_fullsized":"2024-04-26T02:34:53.984785+0000","mapping_epoch":39,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.040487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5776408,"num_objects":706,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11031,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'10945","reported_seq":11124,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:57.302903+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:34:57.302903+0000","last_peered":"2024-04-26T02:34:57.302903+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:34:57.302903+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5067276,"num_objects":619,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10943,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11020","reported_seq":11260,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.750891+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:53.750891+0000","last_peered":"2024-04-26T02:34:53.750891+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:53.750891+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5720850,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10940","reported_seq":11174,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.750155+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:53.750155+0000","last_peered":"2024-04-26T02:34:53.750155+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:53.750155+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5380358,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1241}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11098","reported_seq":11319,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:57.420971+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:34:57.420971+0000","last_peered":"2024-04-26T02:34:57.420971+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:34:57.420971+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5576966,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11097,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11419","reported_seq":11671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:53.941703+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:53.941703+0000","last_peered":"2024-04-26T02:34:53.941703+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:53.941703+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5755404,"num_objects":703,"num_object_clones":0,"num_object_copies":2812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1276}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11215","reported_seq":15430,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:58.064088+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:58.064088+0000","last_peered":"2024-04-26T02:34:58.064088+0000","last_clean":"2024-04-26T02:34:58.064088+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:58.064088+0000","last_undegraded":"2024-04-26T02:34:58.064088+0000","last_fullsized":"2024-04-26T02:34:58.064088+0000","mapping_epoch":35,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5239308,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11133","reported_seq":11367,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:55.839322+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:34:55.839322+0000","last_peered":"2024-04-26T02:34:55.839322+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:34:55.839322+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5435916,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87932096,"num_objects":10741,"num_object_clones":0,"num_object_copies":42964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176816,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10092,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173555712,"data_stored":173555712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689760,"kb_used_data":44432,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93682080,"statfs":{"total":96636764160,"available":95930449920,"internally_reserved":0,"allocated":45498368,"data_stored":45285325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,10,1,6,1,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695756,"kb_used_data":42628,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93676084,"statfs":{"total":96636764160,"available":95924310016,"internally_reserved":0,"allocated":43651072,"data_stored":43438029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572888,"kb_used_data":41488,"kb_used_omap":4,"kb_used_meta":531387,"kb_avail":93798952,"statfs":{"total":96636764160,"available":96050126848,"internally_reserved":0,"allocated":42483712,"data_stored":42270669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":544141007},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697224,"kb_used_data":43648,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93674616,"statfs":{"total":96636764160,"available":95922806784,"internally_reserved":0,"allocated":44695552,"data_stored":44482509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,15,5,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43782144,"data_stored":43782144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42106880,"data_stored":42106880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42565632,"data_stored":42565632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45101056,"data_stored":45101056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:03.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:04.408 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:04.502 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-04-26T02:35:02.743162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80878784,"num_objects":9880,"num_object_clones":0,"num_object_copies":39520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9880,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177677,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10143,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2623464,"kb_used_data":160832,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374863896,"statfs":{"total":386547056640,"available":383860629504,"internally_reserved":0,"allocated":164691968,"data_stored":163839796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2521679411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,10,4,14,12,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19431424,"num_objects":-2372,"num_object_clones":0,"num_object_copies":-9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007970"},"pg_stats":[{"pgid":"1.f","version":"40'10892","reported_seq":11129,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:58.871311+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:34:58.871311+0000","last_peered":"2024-04-26T02:34:58.871311+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:34:58.871311+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":632,"num_object_clones":0,"num_object_copies":2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10892,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11230","reported_seq":13605,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.731146+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:00.731146+0000","last_peered":"2024-04-26T02:35:00.731146+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:00.731146+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928012,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11228,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11025","reported_seq":11196,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.388274+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:02.388274+0000","last_peered":"2024-04-26T02:35:02.388274+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:02.388274+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4954398,"num_objects":606,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11024,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10823","reported_seq":14854,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:58.194280+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:34:58.194280+0000","last_peered":"2024-04-26T02:34:58.194280+0000","last_clean":"2024-04-26T02:34:58.194280+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:34:58.194280+0000","last_undegraded":"2024-04-26T02:34:58.194280+0000","last_fullsized":"2024-04-26T02:34:58.194280+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378572,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10821,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'10952","reported_seq":11188,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:58.960638+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:34:58.960638+0000","last_peered":"2024-04-26T02:34:58.960638+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:34:58.960638+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5280268,"num_objects":645,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1284}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'10980","reported_seq":11241,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:58.992728+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:34:58.992728+0000","last_peered":"2024-04-26T02:34:58.992728+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:34:58.992728+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5079040,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10980,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11149","reported_seq":15407,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:00.613906+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:00.613906+0000","last_peered":"2024-04-26T02:35:00.613906+0000","last_clean":"2024-04-26T02:35:00.613906+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:00.613906+0000","last_undegraded":"2024-04-26T02:35:00.613906+0000","last_fullsized":"2024-04-26T02:35:00.613906+0000","mapping_epoch":33,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4985356,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11409","reported_seq":11669,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.898342+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:00.898342+0000","last_peered":"2024-04-26T02:35:00.898342+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:00.898342+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4747788,"num_objects":580,"num_object_clones":0,"num_object_copies":2320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11102","reported_seq":15197,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:59.141947+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:34:59.141947+0000","last_peered":"2024-04-26T02:34:59.141947+0000","last_clean":"2024-04-26T02:34:59.141947+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:34:59.141947+0000","last_undegraded":"2024-04-26T02:34:59.141947+0000","last_fullsized":"2024-04-26T02:34:59.141947+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:04.110011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5219352,"num_objects":638,"num_object_clones":0,"num_object_copies":2552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11006","reported_seq":11184,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.397926+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:02.397926+0000","last_peered":"2024-04-26T02:35:02.397926+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:02.397926+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4567564,"num_objects":558,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11004,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1437}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11079","reported_seq":11319,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:58.610711+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:34:58.610711+0000","last_peered":"2024-04-26T02:34:58.610711+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:34:58.610711+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5237522,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'10994","reported_seq":11228,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:59.077130+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:34:59.077130+0000","last_peered":"2024-04-26T02:34:59.077130+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:34:59.077130+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4937990,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1295}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11166","reported_seq":11389,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.321104+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:02.321104+0000","last_peered":"2024-04-26T02:35:02.321104+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:02.321104+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5019910,"num_objects":613,"num_object_clones":0,"num_object_copies":2452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11493","reported_seq":11743,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:34:59.189284+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:34:59.189284+0000","last_peered":"2024-04-26T02:34:59.189284+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:34:59.189284+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5149196,"num_objects":629,"num_object_clones":0,"num_object_copies":2516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11215","reported_seq":15430,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:34:58.064088+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:34:58.064088+0000","last_peered":"2024-04-26T02:34:58.064088+0000","last_clean":"2024-04-26T02:34:58.064088+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:34:58.064088+0000","last_undegraded":"2024-04-26T02:34:58.064088+0000","last_fullsized":"2024-04-26T02:34:58.064088+0000","mapping_epoch":35,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5239308,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11189","reported_seq":11423,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.747330+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:00.747330+0000","last_peered":"2024-04-26T02:35:00.747330+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:00.747330+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4977164,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80878784,"num_objects":9880,"num_object_clones":0,"num_object_copies":39520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9880,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177677,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10143,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":161857536,"data_stored":161857536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690132,"kb_used_data":40652,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93681708,"statfs":{"total":96636764160,"available":95930068992,"internally_reserved":0,"allocated":41627648,"data_stored":41414605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,10,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679448,"kb_used_data":38792,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93692392,"statfs":{"total":96636764160,"available":95941009408,"internally_reserved":0,"allocated":39723008,"data_stored":39509965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572888,"kb_used_data":41488,"kb_used_omap":4,"kb_used_meta":531387,"kb_avail":93798952,"statfs":{"total":96636764160,"available":96050126848,"internally_reserved":0,"allocated":42483712,"data_stored":42270669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":544141007},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680996,"kb_used_data":39900,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93690844,"statfs":{"total":96636764160,"available":95939424256,"internally_reserved":0,"allocated":40857600,"data_stored":40644557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,0,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":39866368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42106880,"data_stored":42106880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38666240,"data_stored":38666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41218048,"data_stored":41218048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:04.504 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:05.955 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:05.956 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:05.992 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-04-26T02:35:04.743684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76553408,"num_objects":9352,"num_object_clones":0,"num_object_copies":37408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178205,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10143,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625600,"kb_used_data":153240,"kb_used_omap":14,"kb_used_meta":2472305,"kb_avail":374861760,"statfs":{"total":386547056640,"available":383858442240,"internally_reserved":0,"allocated":156917760,"data_stored":156065588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2531640883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,5,14,6,9,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20504576,"num_objects":-2503,"num_object_clones":0,"num_object_copies":-10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003017"},"pg_stats":[{"pgid":"1.f","version":"40'10948","reported_seq":11185,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:03.996839+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:03.996839+0000","last_peered":"2024-04-26T02:35:03.996839+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:03.996839+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":576,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11230","reported_seq":13605,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.731146+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:00.731146+0000","last_peered":"2024-04-26T02:35:00.731146+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:00.731146+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4928012,"num_objects":602,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11228,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11025","reported_seq":11196,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.388274+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:02.388274+0000","last_peered":"2024-04-26T02:35:02.388274+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:02.388274+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4954398,"num_objects":606,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11024,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10885","reported_seq":14916,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:03.120014+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:03.120014+0000","last_peered":"2024-04-26T02:35:03.120014+0000","last_clean":"2024-04-26T02:35:03.120014+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:03.120014+0000","last_undegraded":"2024-04-26T02:35:03.120014+0000","last_fullsized":"2024-04-26T02:35:03.120014+0000","mapping_epoch":37,"log_start":"28'800","ondisk_log_start":"28'800","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:08.290561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4870668,"num_objects":595,"num_object_clones":0,"num_object_copies":2380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11024","reported_seq":11256,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.137859+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:04.137859+0000","last_peered":"2024-04-26T02:35:04.137859+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:04.137859+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4690444,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11023","reported_seq":11284,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.014094+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:04.014094+0000","last_peered":"2024-04-26T02:35:04.014094+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:04.014094+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11149","reported_seq":15407,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:00.613906+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:00.613906+0000","last_peered":"2024-04-26T02:35:00.613906+0000","last_clean":"2024-04-26T02:35:00.613906+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:00.613906+0000","last_undegraded":"2024-04-26T02:35:00.613906+0000","last_fullsized":"2024-04-26T02:35:00.613906+0000","mapping_epoch":33,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4985356,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11409","reported_seq":11669,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.898342+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:00.898342+0000","last_peered":"2024-04-26T02:35:00.898342+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:00.898342+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4747788,"num_objects":580,"num_object_clones":0,"num_object_copies":2320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11152","reported_seq":15248,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:04.063087+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:04.063087+0000","last_peered":"2024-04-26T02:35:04.063087+0000","last_clean":"2024-04-26T02:35:04.063087+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:04.063087+0000","last_undegraded":"2024-04-26T02:35:04.063087+0000","last_fullsized":"2024-04-26T02:35:04.063087+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4809752,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11006","reported_seq":11184,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.397926+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:02.397926+0000","last_peered":"2024-04-26T02:35:02.397926+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:02.397926+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4567564,"num_objects":558,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11004,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1437}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11131","reported_seq":11371,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.061353+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:04.061353+0000","last_peered":"2024-04-26T02:35:04.061353+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:04.061353+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4811538,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11065","reported_seq":11299,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.290886+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:04.290886+0000","last_peered":"2024-04-26T02:35:04.290886+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:04.290886+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4356358,"num_objects":532,"num_object_clones":0,"num_object_copies":2128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11064,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1366}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11166","reported_seq":11389,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:02.321104+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:02.321104+0000","last_peered":"2024-04-26T02:35:02.321104+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:02.321104+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5019910,"num_objects":613,"num_object_clones":0,"num_object_copies":2452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11556","reported_seq":11803,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.109351+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:04.109351+0000","last_peered":"2024-04-26T02:35:04.109351+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:04.109351+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4633100,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11554,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1413}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11274","reported_seq":15489,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:03.342245+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:03.342245+0000","last_peered":"2024-04-26T02:35:03.342245+0000","last_clean":"2024-04-26T02:35:03.342245+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:03.342245+0000","last_undegraded":"2024-04-26T02:35:03.342245+0000","last_fullsized":"2024-04-26T02:35:03.342245+0000","mapping_epoch":35,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:08.290530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4755980,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11189","reported_seq":11423,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:00.747330+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:00.747330+0000","last_peered":"2024-04-26T02:35:00.747330+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:00.747330+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4977164,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":76553408,"num_objects":9352,"num_object_clones":0,"num_object_copies":37408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178205,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10143,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":153931776,"data_stored":153931776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690524,"kb_used_data":36884,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93681316,"statfs":{"total":96636764160,"available":95929667584,"internally_reserved":0,"allocated":37769216,"data_stored":37556173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679448,"kb_used_data":38792,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93692392,"statfs":{"total":96636764160,"available":95941009408,"internally_reserved":0,"allocated":39723008,"data_stored":39509965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574632,"kb_used_data":37664,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93797208,"statfs":{"total":96636764160,"available":96048340992,"internally_reserved":0,"allocated":38567936,"data_stored":38354893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,6,0,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680996,"kb_used_data":39900,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93690844,"statfs":{"total":96636764160,"available":95939424256,"internally_reserved":0,"allocated":40857600,"data_stored":40644557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656469711},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,0,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39866368,"data_stored":39866368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38109184,"data_stored":38109184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38666240,"data_stored":38666240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37289984,"data_stored":37289984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:05.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:05.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'10948", 'reported_seq': 11185, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:03.996839+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:03.996839+0000', 'last_peered': '2024-04-26T02:35:03.996839+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:03.996839+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'900", 'ondisk_log_start': "28'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4718592, 'num_objects': 576, 'num_object_clones': 0, 'num_object_copies': 2304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10948, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11230", 'reported_seq': 13605, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:00.731146+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:00.731146+0000', 'last_peered': '2024-04-26T02:35:00.731146+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:00.731146+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4928012, 'num_objects': 602, 'num_object_clones': 0, 'num_object_copies': 2408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 930, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11228, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1150, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11025", 'reported_seq': 11196, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:02.388274+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:02.388274+0000', 'last_peered': '2024-04-26T02:35:02.388274+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:02.388274+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4954398, 'num_objects': 606, 'num_object_clones': 0, 'num_object_copies': 2424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 606, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11024, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1378}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11024", 'reported_seq': 11256, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:04.137859+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:04.137859+0000', 'last_peered': '2024-04-26T02:35:04.137859+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:04.137859+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4690444, 'num_objects': 573, 'num_object_clones': 0, 'num_object_copies': 2292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11022, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:05.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11023", 'reported_seq': 11284, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:04.014094+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:04.014094+0000', 'last_peered': '2024-04-26T02:35:04.014094+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:04.014094+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4726784, 'num_objects': 577, 'num_object_clones': 0, 'num_object_copies': 2308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11023, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1373}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11409", 'reported_seq': 11669, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:00.898342+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:00.898342+0000', 'last_peered': '2024-04-26T02:35:00.898342+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:00.898342+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4747788, 'num_objects': 580, 'num_object_clones': 0, 'num_object_copies': 2320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11407, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11006", 'reported_seq': 11184, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:02.397926+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:02.397926+0000', 'last_peered': '2024-04-26T02:35:02.397926+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:02.397926+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4567564, 'num_objects': 558, 'num_object_clones': 0, 'num_object_copies': 2232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11004, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1437}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11131", 'reported_seq': 11371, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:04.061353+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:04.061353+0000', 'last_peered': '2024-04-26T02:35:04.061353+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:04.061353+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4811538, 'num_objects': 588, 'num_object_clones': 0, 'num_object_copies': 2352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11128, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11065", 'reported_seq': 11299, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:04.290886+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:04.290886+0000', 'last_peered': '2024-04-26T02:35:04.290886+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:04.290886+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4356358, 'num_objects': 532, 'num_object_clones': 0, 'num_object_copies': 2128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11064, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:05.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11166", 'reported_seq': 11389, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:02.321104+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:02.321104+0000', 'last_peered': '2024-04-26T02:35:02.321104+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:02.321104+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5019910, 'num_objects': 613, 'num_object_clones': 0, 'num_object_copies': 2452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11165, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:05.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:05.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11556", 'reported_seq': 11803, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:04.109351+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:04.109351+0000', 'last_peered': '2024-04-26T02:35:04.109351+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:04.109351+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4633100, 'num_objects': 566, 'num_object_clones': 0, 'num_object_copies': 2264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11554, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:05.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:05.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11189", 'reported_seq': 11423, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:00.747330+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:00.747330+0000', 'last_peered': '2024-04-26T02:35:00.747330+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:00.747330+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4977164, 'num_objects': 608, 'num_object_clones': 0, 'num_object_copies': 2432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11187, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:05.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:07.829 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22559/36556 objects degraded (61.711%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":704,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9139,"data_bytes":74808512,"bytes_used":2688974848,"bytes_avail":383858081792,"bytes_total":386547056640,"degraded_objects":22559,"degraded_total":36556,"degraded_ratio":0.6171079986869461,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:35:07.908 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:07.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:09.199 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:09.199 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:09.248 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-04-26T02:35:08.748821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":72211648,"num_objects":8822,"num_object_clones":0,"num_object_copies":35288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178735,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10210,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626724,"kb_used_data":141868,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374860636,"statfs":{"total":386547056640,"available":383857291264,"internally_reserved":0,"allocated":145272832,"data_stored":144420660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2544420403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,8,12,10,8,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18112512,"num_objects":-2211,"num_object_clones":0,"num_object_copies":-8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007171"},"pg_stats":[{"pgid":"1.f","version":"40'10948","reported_seq":11185,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:03.996839+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:03.996839+0000","last_peered":"2024-04-26T02:35:03.996839+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:03.996839+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":576,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11285","reported_seq":13797,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:05.975228+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:05.975228+0000","last_peered":"2024-04-26T02:35:05.975228+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:05.975228+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4477452,"num_objects":547,"num_object_clones":0,"num_object_copies":2188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11083","reported_seq":11258,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:07.501382+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:07.501382+0000","last_peered":"2024-04-26T02:35:07.501382+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:07.501382+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4479262,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11082,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1436}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10951","reported_seq":14982,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:08.264281+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:08.264281+0000","last_peered":"2024-04-26T02:35:08.264281+0000","last_clean":"2024-04-26T02:35:08.264281+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:08.264281+0000","last_undegraded":"2024-04-26T02:35:08.264281+0000","last_fullsized":"2024-04-26T02:35:08.264281+0000","mapping_epoch":37,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4329996,"num_objects":529,"num_object_clones":0,"num_object_copies":2116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11024","reported_seq":11256,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.137859+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:04.137859+0000","last_peered":"2024-04-26T02:35:04.137859+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:04.137859+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4690444,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11023","reported_seq":11284,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.014094+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:04.014094+0000","last_peered":"2024-04-26T02:35:04.014094+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:04.014094+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11207","reported_seq":15465,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:05.873109+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:05.873109+0000","last_peered":"2024-04-26T02:35:05.873109+0000","last_clean":"2024-04-26T02:35:05.873109+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:05.873109+0000","last_undegraded":"2024-04-26T02:35:05.873109+0000","last_fullsized":"2024-04-26T02:35:05.873109+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:06.113692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4510220,"num_objects":551,"num_object_clones":0,"num_object_copies":2204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11464","reported_seq":11723,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:05.918698+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:05.918698+0000","last_peered":"2024-04-26T02:35:05.918698+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:05.918698+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297228,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11462,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11152","reported_seq":15248,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:04.063087+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:04.063087+0000","last_peered":"2024-04-26T02:35:04.063087+0000","last_clean":"2024-04-26T02:35:04.063087+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:04.063087+0000","last_undegraded":"2024-04-26T02:35:04.063087+0000","last_fullsized":"2024-04-26T02:35:04.063087+0000","mapping_epoch":39,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4809752,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11061","reported_seq":11240,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:07.502558+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:07.502558+0000","last_peered":"2024-04-26T02:35:07.502558+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:07.502558+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4117004,"num_objects":503,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11131","reported_seq":11371,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.061353+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:04.061353+0000","last_peered":"2024-04-26T02:35:04.061353+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:04.061353+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4811538,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11065","reported_seq":11299,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.290886+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:04.290886+0000","last_peered":"2024-04-26T02:35:04.290886+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:04.290886+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4356358,"num_objects":532,"num_object_clones":0,"num_object_copies":2128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11064,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1366}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11243","reported_seq":11464,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:07.372957+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:07.372957+0000","last_peered":"2024-04-26T02:35:07.372957+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:07.372957+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4389126,"num_objects":536,"num_object_clones":0,"num_object_copies":2144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11242,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11556","reported_seq":11803,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:04.109351+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:04.109351+0000","last_peered":"2024-04-26T02:35:04.109351+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:04.109351+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4633100,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11554,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1413}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11335","reported_seq":15549,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:08.379108+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:08.379108+0000","last_peered":"2024-04-26T02:35:08.379108+0000","last_clean":"2024-04-26T02:35:08.379108+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:08.379108+0000","last_undegraded":"2024-04-26T02:35:08.379108+0000","last_fullsized":"2024-04-26T02:35:08.379108+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:13.378731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4256268,"num_objects":520,"num_object_clones":0,"num_object_copies":2080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11234","reported_seq":11468,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:05.725156+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:05.725156+0000","last_peered":"2024-04-26T02:35:05.725156+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:05.725156+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4608524,"num_objects":563,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11232,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":72211648,"num_objects":8822,"num_object_clones":0,"num_object_copies":35288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178735,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10210,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":142307328,"data_stored":142307328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690524,"kb_used_data":36884,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93681316,"statfs":{"total":96636764160,"available":95929667584,"internally_reserved":0,"allocated":37769216,"data_stored":37556173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679860,"kb_used_data":35044,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93691980,"statfs":{"total":96636764160,"available":95940587520,"internally_reserved":0,"allocated":35885056,"data_stored":35672013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,1,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574992,"kb_used_data":33856,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93796848,"statfs":{"total":96636764160,"available":96047972352,"internally_reserved":0,"allocated":34668544,"data_stored":34455501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681348,"kb_used_data":36084,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93690492,"statfs":{"total":96636764160,"available":95939063808,"internally_reserved":0,"allocated":36950016,"data_stored":36736973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35975168,"data_stored":35975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34263040,"data_stored":34263040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34779136,"data_stored":34779136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37289984,"data_stored":37289984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:12.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:13.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:13.659 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:13.919 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-04-26T02:35:12.749818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65625280,"num_objects":8018,"num_object_clones":0,"num_object_copies":32072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179539,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10278,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2615580,"kb_used_data":130716,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374871780,"statfs":{"total":386547056640,"available":383868702720,"internally_reserved":0,"allocated":133853184,"data_stored":133001012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2544420403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,9,12,7,13,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18808832,"num_objects":-2296,"num_object_clones":0,"num_object_copies":-9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007158"},"pg_stats":[{"pgid":"1.f","version":"40'11002","reported_seq":11239,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.312815+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:09.312815+0000","last_peered":"2024-04-26T02:35:09.312815+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:09.312815+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11337","reported_seq":13985,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:11.046928+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:11.046928+0000","last_peered":"2024-04-26T02:35:11.046928+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:11.046928+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4051468,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1285,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11132","reported_seq":11307,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.384816+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:12.384816+0000","last_peered":"2024-04-26T02:35:12.384816+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:12.384816+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4077854,"num_objects":499,"num_object_clones":0,"num_object_copies":1996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11131,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1485}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'10951","reported_seq":14982,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:08.264281+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:08.264281+0000","last_peered":"2024-04-26T02:35:08.264281+0000","last_clean":"2024-04-26T02:35:08.264281+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:08.264281+0000","last_undegraded":"2024-04-26T02:35:08.264281+0000","last_fullsized":"2024-04-26T02:35:08.264281+0000","mapping_epoch":37,"log_start":"28'900","ondisk_log_start":"28'900","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4329996,"num_objects":529,"num_object_clones":0,"num_object_copies":2116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11088","reported_seq":11323,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.333624+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:09.333624+0000","last_peered":"2024-04-26T02:35:09.333624+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:09.333624+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4166156,"num_objects":509,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1420}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11077","reported_seq":11338,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.155059+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:09.155059+0000","last_peered":"2024-04-26T02:35:09.155059+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:09.155059+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4284416,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11077,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11273","reported_seq":15529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:10.946608+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:10.946608+0000","last_peered":"2024-04-26T02:35:10.946608+0000","last_clean":"2024-04-26T02:35:10.946608+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:10.946608+0000","last_undegraded":"2024-04-26T02:35:10.946608+0000","last_fullsized":"2024-04-26T02:35:10.946608+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:11.163732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3969548,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11271,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11525","reported_seq":11784,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:11.048840+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:11.048840+0000","last_peered":"2024-04-26T02:35:11.048840+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:11.048840+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3797516,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11224","reported_seq":15319,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:09.252475+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:09.252475+0000","last_peered":"2024-04-26T02:35:09.252475+0000","last_clean":"2024-04-26T02:35:09.252475+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:09.252475+0000","last_undegraded":"2024-04-26T02:35:09.252475+0000","last_fullsized":"2024-04-26T02:35:09.252475+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:10.064167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4219928,"num_objects":516,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11221,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11127","reported_seq":11306,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.523147+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:12.523147+0000","last_peered":"2024-04-26T02:35:12.523147+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:12.523147+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576332,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11190","reported_seq":11430,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.293824+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:09.293824+0000","last_peered":"2024-04-26T02:35:09.293824+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:09.293824+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4328210,"num_objects":529,"num_object_clones":0,"num_object_copies":2116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11107","reported_seq":11342,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.083287+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:09.083287+0000","last_peered":"2024-04-26T02:35:09.083287+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:09.083287+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4012294,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11296","reported_seq":11519,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.460188+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:12.460188+0000","last_peered":"2024-04-26T02:35:12.460188+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:12.460188+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954950,"num_objects":483,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11608","reported_seq":11859,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:09.116423+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:09.116423+0000","last_peered":"2024-04-26T02:35:09.116423+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:09.116423+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4207116,"num_objects":514,"num_object_clones":0,"num_object_copies":2056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11335","reported_seq":15549,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:08.379108+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:08.379108+0000","last_peered":"2024-04-26T02:35:08.379108+0000","last_clean":"2024-04-26T02:35:08.379108+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:08.379108+0000","last_undegraded":"2024-04-26T02:35:08.379108+0000","last_fullsized":"2024-04-26T02:35:08.379108+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:13.378731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4256268,"num_objects":520,"num_object_clones":0,"num_object_copies":2080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11294","reported_seq":11528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:10.905844+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:10.905844+0000","last_peered":"2024-04-26T02:35:10.905844+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:10.905844+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4117004,"num_objects":503,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11292,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":65625280,"num_objects":8018,"num_object_clones":0,"num_object_copies":32072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179539,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10278,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130633728,"data_stored":130633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678536,"kb_used_data":33216,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93693304,"statfs":{"total":96636764160,"available":95941943296,"internally_reserved":0,"allocated":34013184,"data_stored":33800141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,7,1,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680196,"kb_used_data":31212,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93691644,"statfs":{"total":96636764160,"available":95940243456,"internally_reserved":0,"allocated":31961088,"data_stored":31748045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574992,"kb_used_data":33856,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93796848,"statfs":{"total":96636764160,"available":96047972352,"internally_reserved":0,"allocated":34668544,"data_stored":34455501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681856,"kb_used_data":32432,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93689984,"statfs":{"total":96636764160,"available":95938543616,"internally_reserved":0,"allocated":33210368,"data_stored":32997325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32067584,"data_stored":32067584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34263040,"data_stored":34263040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30908416,"data_stored":30908416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33394688,"data_stored":33394688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:13.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:15.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:15.439 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:15.572 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-04-26T02:35:14.750302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":61119680,"num_objects":7468,"num_object_clones":0,"num_object_copies":29872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180089,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10278,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160916,"ondisk_log_size":160916,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2616252,"kb_used_data":123076,"kb_used_omap":14,"kb_used_meta":2493105,"kb_avail":374871108,"statfs":{"total":386547056640,"available":383868014592,"internally_reserved":0,"allocated":126029824,"data_stored":125177652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2552940083},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,9,16,14,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007139"},"pg_stats":[{"pgid":"1.f","version":"40'11057","reported_seq":11294,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.177201+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:14.177201+0000","last_peered":"2024-04-26T02:35:14.177201+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:14.177201+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3825664,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11337","reported_seq":13985,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:11.046928+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:11.046928+0000","last_peered":"2024-04-26T02:35:11.046928+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:11.046928+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4051468,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1285,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11132","reported_seq":11307,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.384816+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:12.384816+0000","last_peered":"2024-04-26T02:35:12.384816+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:12.384816+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4077854,"num_objects":499,"num_object_clones":0,"num_object_copies":1996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11131,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1485}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11012","reported_seq":15043,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:13.410820+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:13.410820+0000","last_peered":"2024-04-26T02:35:13.410820+0000","last_clean":"2024-04-26T02:35:13.410820+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:13.410820+0000","last_undegraded":"2024-04-26T02:35:13.410820+0000","last_fullsized":"2024-04-26T02:35:13.410820+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:18.442208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3830284,"num_objects":468,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11160","reported_seq":11396,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.212267+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:14.212267+0000","last_peered":"2024-04-26T02:35:14.212267+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:14.212267+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576332,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11147","reported_seq":11408,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.362847+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:14.362847+0000","last_peered":"2024-04-26T02:35:14.362847+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:14.362847+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3710976,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11273","reported_seq":15529,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:10.946608+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:10.946608+0000","last_peered":"2024-04-26T02:35:10.946608+0000","last_clean":"2024-04-26T02:35:10.946608+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:10.946608+0000","last_undegraded":"2024-04-26T02:35:10.946608+0000","last_fullsized":"2024-04-26T02:35:10.946608+0000","mapping_epoch":33,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:11.163732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3969548,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11271,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11525","reported_seq":11784,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:11.048840+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:11.048840+0000","last_peered":"2024-04-26T02:35:11.048840+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:11.048840+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3797516,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11279","reported_seq":15375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:14.356298+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:14.356298+0000","last_peered":"2024-04-26T02:35:14.356298+0000","last_clean":"2024-04-26T02:35:14.356298+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:14.356298+0000","last_undegraded":"2024-04-26T02:35:14.356298+0000","last_fullsized":"2024-04-26T02:35:14.356298+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:15.976040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3769368,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11127","reported_seq":11306,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.523147+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:12.523147+0000","last_peered":"2024-04-26T02:35:12.523147+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:12.523147+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576332,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11125,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11240","reported_seq":11480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.233889+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:14.233889+0000","last_peered":"2024-04-26T02:35:14.233889+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:14.233889+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3918610,"num_objects":479,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11176","reported_seq":11411,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.027302+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:14.027302+0000","last_peered":"2024-04-26T02:35:14.027302+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:14.027302+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447046,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1477}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11296","reported_seq":11519,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:12.460188+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:12.460188+0000","last_peered":"2024-04-26T02:35:12.460188+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:12.460188+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954950,"num_objects":483,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11668","reported_seq":11920,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.212714+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:14.212714+0000","last_peered":"2024-04-26T02:35:14.212714+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:14.212714+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3715596,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11393","reported_seq":15607,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:13.345890+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:13.345890+0000","last_peered":"2024-04-26T02:35:13.345890+0000","last_clean":"2024-04-26T02:35:13.345890+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:13.345890+0000","last_undegraded":"2024-04-26T02:35:13.345890+0000","last_fullsized":"2024-04-26T02:35:13.345890+0000","mapping_epoch":35,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:18.442185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781132,"num_objects":462,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11294","reported_seq":11528,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:10.905844+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:10.905844+0000","last_peered":"2024-04-26T02:35:10.905844+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:10.905844+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4117004,"num_objects":503,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11292,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":61119680,"num_objects":7468,"num_object_clones":0,"num_object_copies":29872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180089,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10278,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122826752,"data_stored":122826752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160916,"ondisk_log_size":160916,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678868,"kb_used_data":29388,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93692972,"statfs":{"total":96636764160,"available":95941603328,"internally_reserved":0,"allocated":30093312,"data_stored":29880269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,12,5,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680196,"kb_used_data":31212,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93691644,"statfs":{"total":96636764160,"available":95940243456,"internally_reserved":0,"allocated":31961088,"data_stored":31748045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575332,"kb_used_data":30044,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93796508,"statfs":{"total":96636764160,"available":96047624192,"internally_reserved":0,"allocated":30765056,"data_stored":30552013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681856,"kb_used_data":32432,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93689984,"statfs":{"total":96636764160,"available":95938543616,"internally_reserved":0,"allocated":33210368,"data_stored":32997325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":664989391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32067584,"data_stored":32067584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30355456,"data_stored":30355456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30908416,"data_stored":30908416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29495296,"data_stored":29495296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11057", 'reported_seq': 11294, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.177201+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:14.177201+0000', 'last_peered': '2024-04-26T02:35:14.177201+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:14.177201+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1000", 'ondisk_log_start': "28'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3825664, 'num_objects': 467, 'num_object_clones': 0, 'num_object_copies': 1868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11057, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1460}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11337", 'reported_seq': 13985, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:11.046928+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:11.046928+0000', 'last_peered': '2024-04-26T02:35:11.046928+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:11.046928+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4051468, 'num_objects': 495, 'num_object_clones': 0, 'num_object_copies': 1980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 795, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11335, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1285, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11132", 'reported_seq': 11307, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:12.384816+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:12.384816+0000', 'last_peered': '2024-04-26T02:35:12.384816+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:12.384816+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4077854, 'num_objects': 499, 'num_object_clones': 0, 'num_object_copies': 1996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 499, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11131, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1485}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:15.573 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11160", 'reported_seq': 11396, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.212267+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:14.212267+0000', 'last_peered': '2024-04-26T02:35:14.212267+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:14.212267+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3576332, 'num_objects': 437, 'num_object_clones': 0, 'num_object_copies': 1748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11158, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1492}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11147", 'reported_seq': 11408, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.362847+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:14.362847+0000', 'last_peered': '2024-04-26T02:35:14.362847+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:14.362847+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3710976, 'num_objects': 453, 'num_object_clones': 0, 'num_object_copies': 1812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11147, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11525", 'reported_seq': 11784, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:11.048840+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:11.048840+0000', 'last_peered': '2024-04-26T02:35:11.048840+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:11.048840+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3797516, 'num_objects': 464, 'num_object_clones': 0, 'num_object_copies': 1856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11523, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11127", 'reported_seq': 11306, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:12.523147+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:12.523147+0000', 'last_peered': '2024-04-26T02:35:12.523147+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:12.523147+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3576332, 'num_objects': 437, 'num_object_clones': 0, 'num_object_copies': 1748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11125, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1558}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11240", 'reported_seq': 11480, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.233889+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:14.233889+0000', 'last_peered': '2024-04-26T02:35:14.233889+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:14.233889+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3918610, 'num_objects': 479, 'num_object_clones': 0, 'num_object_copies': 1916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11237, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1455}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:15.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11176", 'reported_seq': 11411, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.027302+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:14.027302+0000', 'last_peered': '2024-04-26T02:35:14.027302+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:14.027302+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3447046, 'num_objects': 421, 'num_object_clones': 0, 'num_object_copies': 1684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11175, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11296", 'reported_seq': 11519, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:12.460188+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:12.460188+0000', 'last_peered': '2024-04-26T02:35:12.460188+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:12.460188+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954950, 'num_objects': 483, 'num_object_clones': 0, 'num_object_copies': 1932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11668", 'reported_seq': 11920, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:14.212714+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:14.212714+0000', 'last_peered': '2024-04-26T02:35:14.212714+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:14.212714+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3715596, 'num_objects': 454, 'num_object_clones': 0, 'num_object_copies': 1816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11666, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:15.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11294", 'reported_seq': 11528, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:10.905844+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:10.905844+0000', 'last_peered': '2024-04-26T02:35:10.905844+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:10.905844+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4117004, 'num_objects': 503, 'num_object_clones': 0, 'num_object_copies': 2012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11292, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1484}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:15.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:17.169 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:17.169 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22440/28900 objects degraded (77.647%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":713,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7225,"data_bytes":59129024,"bytes_used":2679328768,"bytes_avail":383867727872,"bytes_total":386547056640,"degraded_objects":22440,"degraded_total":28900,"degraded_ratio":0.77647058823529425,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:35:17.276 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:17.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:18.772 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:18.772 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:18.834 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-04-26T02:35:18.758539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56655040,"num_objects":6923,"num_object_clones":0,"num_object_copies":27692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180634,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10329,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160961,"ondisk_log_size":160961,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2604176,"kb_used_data":111020,"kb_used_omap":14,"kb_used_meta":2493105,"kb_avail":374883184,"statfs":{"total":386547056640,"available":383880380416,"internally_reserved":0,"allocated":113684480,"data_stored":112832308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2552940083},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,6,8,16,15,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18153472,"num_objects":-2216,"num_object_clones":0,"num_object_copies":-8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014343"},"pg_stats":[{"pgid":"1.f","version":"40'11057","reported_seq":11294,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.177201+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:14.177201+0000","last_peered":"2024-04-26T02:35:14.177201+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:14.177201+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3825664,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1460}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11411","reported_seq":14161,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:16.177620+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:16.177620+0000","last_peered":"2024-04-26T02:35:16.177620+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:16.177620+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445260,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11409,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11199","reported_seq":11374,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:17.585564+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:17.585564+0000","last_peered":"2024-04-26T02:35:17.585564+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:17.585564+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3528990,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11198,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1552}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11066","reported_seq":15097,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:18.390173+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:18.390173+0000","last_peered":"2024-04-26T02:35:18.390173+0000","last_clean":"2024-04-26T02:35:18.390173+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:18.390173+0000","last_undegraded":"2024-04-26T02:35:18.390173+0000","last_fullsized":"2024-04-26T02:35:18.390173+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3387916,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11064,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11160","reported_seq":11396,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.212267+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:14.212267+0000","last_peered":"2024-04-26T02:35:14.212267+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:14.212267+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3576332,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11147","reported_seq":11408,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.362847+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:14.362847+0000","last_peered":"2024-04-26T02:35:14.362847+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:14.362847+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3710976,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11338","reported_seq":15596,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:15.862273+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:15.862273+0000","last_peered":"2024-04-26T02:35:15.862273+0000","last_clean":"2024-04-26T02:35:15.862273+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:15.862273+0000","last_undegraded":"2024-04-26T02:35:15.862273+0000","last_fullsized":"2024-04-26T02:35:15.862273+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:17.115035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437068,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11583","reported_seq":11841,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:15.956179+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:15.956179+0000","last_peered":"2024-04-26T02:35:15.956179+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:15.956179+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3322380,"num_objects":406,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11279","reported_seq":15375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:14.356298+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:14.356298+0000","last_peered":"2024-04-26T02:35:14.356298+0000","last_clean":"2024-04-26T02:35:14.356298+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:14.356298+0000","last_undegraded":"2024-04-26T02:35:14.356298+0000","last_fullsized":"2024-04-26T02:35:14.356298+0000","mapping_epoch":39,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:15.976040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3769368,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11181","reported_seq":11360,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:17.408180+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:17.408180+0000","last_peered":"2024-04-26T02:35:17.408180+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:17.408180+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3133964,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1612}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11240","reported_seq":11480,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.233889+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:14.233889+0000","last_peered":"2024-04-26T02:35:14.233889+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:14.233889+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3918610,"num_objects":479,"num_object_clones":0,"num_object_copies":1916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11176","reported_seq":11411,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.027302+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:14.027302+0000","last_peered":"2024-04-26T02:35:14.027302+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:14.027302+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447046,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1477}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11360","reported_seq":11583,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:17.527000+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:17.527000+0000","last_peered":"2024-04-26T02:35:17.527000+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:17.527000+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3430662,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1626}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11668","reported_seq":11920,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:14.212714+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:14.212714+0000","last_peered":"2024-04-26T02:35:14.212714+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:14.212714+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3715596,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11456","reported_seq":15671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:18.553153+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:18.553153+0000","last_peered":"2024-04-26T02:35:18.553153+0000","last_clean":"2024-04-26T02:35:18.553153+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:18.553153+0000","last_undegraded":"2024-04-26T02:35:18.553153+0000","last_fullsized":"2024-04-26T02:35:18.553153+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3265036,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11340","reported_seq":11569,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:16.112597+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:16.112597+0000","last_peered":"2024-04-26T02:35:16.112597+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:16.112597+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3740172,"num_objects":457,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":56655040,"num_objects":6923,"num_object_clones":0,"num_object_copies":27692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180634,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10329,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":111034368,"data_stored":111034368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160961,"ondisk_log_size":160961,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678868,"kb_used_data":29388,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93692972,"statfs":{"total":96636764160,"available":95941603328,"internally_reserved":0,"allocated":30093312,"data_stored":29880269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,12,5,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684020,"kb_used_data":26732,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93687820,"statfs":{"total":96636764160,"available":95936327680,"internally_reserved":0,"allocated":27373568,"data_stored":27160525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":673053125},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559152,"kb_used_data":26344,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93812688,"statfs":{"total":96636764160,"available":96064192512,"internally_reserved":0,"allocated":26976256,"data_stored":26763213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682136,"kb_used_data":28556,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93689704,"statfs":{"total":96636764160,"available":95938256896,"internally_reserved":0,"allocated":29241344,"data_stored":29028301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669249231},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28135424,"data_stored":28135424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26411008,"data_stored":26411008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26992640,"data_stored":26992640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29495296,"data_stored":29495296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:21.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:23.561 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:23.561 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:23.680 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-04-26T02:35:22.759546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49675456,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181486,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10375,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2575332,"kb_used_data":98812,"kb_used_omap":14,"kb_used_meta":2476465,"kb_avail":374912028,"statfs":{"total":386547056640,"available":383909916672,"internally_reserved":0,"allocated":101183488,"data_stored":100331316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2535900723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,9,10,21,8,14,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19283968,"num_objects":-2354,"num_object_clones":0,"num_object_copies":-9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010215"},"pg_stats":[{"pgid":"1.f","version":"40'11122","reported_seq":11359,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:19.427405+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:19.427405+0000","last_peered":"2024-04-26T02:35:19.427405+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:19.427405+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3293184,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11467","reported_seq":14308,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.215870+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:21.215870+0000","last_peered":"2024-04-26T02:35:21.215870+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:21.215870+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2986508,"num_objects":365,"num_object_clones":0,"num_object_copies":1460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11258","reported_seq":11433,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.397883+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:22.397883+0000","last_peered":"2024-04-26T02:35:22.397883+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:22.397883+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3045662,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11257,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1611}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11066","reported_seq":15097,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:18.390173+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:18.390173+0000","last_peered":"2024-04-26T02:35:18.390173+0000","last_clean":"2024-04-26T02:35:18.390173+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:18.390173+0000","last_undegraded":"2024-04-26T02:35:18.390173+0000","last_fullsized":"2024-04-26T02:35:18.390173+0000","mapping_epoch":37,"log_start":"28'1000","ondisk_log_start":"28'1000","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3387916,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11064,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11215","reported_seq":11451,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:19.354893+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:19.354893+0000","last_peered":"2024-04-26T02:35:19.354893+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:19.354893+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3125772,"num_objects":382,"num_object_clones":0,"num_object_copies":1528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11208","reported_seq":11469,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:19.492050+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:19.492050+0000","last_peered":"2024-04-26T02:35:19.492050+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:19.492050+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":392,"num_object_clones":0,"num_object_copies":1568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11395","reported_seq":15653,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:21.052186+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:21.052186+0000","last_peered":"2024-04-26T02:35:21.052186+0000","last_clean":"2024-04-26T02:35:21.052186+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:21.052186+0000","last_undegraded":"2024-04-26T02:35:21.052186+0000","last_fullsized":"2024-04-26T02:35:21.052186+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:22.166215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2970124,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11649","reported_seq":11909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.164009+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:21.164009+0000","last_peered":"2024-04-26T02:35:21.164009+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:21.164009+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2781708,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11339","reported_seq":15435,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:19.222718+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:19.222718+0000","last_peered":"2024-04-26T02:35:19.222718+0000","last_clean":"2024-04-26T02:35:19.222718+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:19.222718+0000","last_undegraded":"2024-04-26T02:35:19.222718+0000","last_fullsized":"2024-04-26T02:35:19.222718+0000","mapping_epoch":39,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:21.072759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3277848,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11248","reported_seq":11425,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.672884+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:22.672884+0000","last_peered":"2024-04-26T02:35:22.672884+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:22.672884+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585100,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11319","reported_seq":11559,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:19.433618+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:19.433618+0000","last_peered":"2024-04-26T02:35:19.433618+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:19.433618+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3271442,"num_objects":400,"num_object_clones":0,"num_object_copies":1600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11223","reported_seq":11458,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:18.983745+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:18.983745+0000","last_peered":"2024-04-26T02:35:18.983745+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:18.983745+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3062022,"num_objects":374,"num_object_clones":0,"num_object_copies":1496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1524}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11411","reported_seq":11634,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.671981+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:22.671981+0000","last_peered":"2024-04-26T02:35:22.671981+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:22.671981+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3012870,"num_objects":368,"num_object_clones":0,"num_object_copies":1472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11721","reported_seq":11974,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:19.473959+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:19.473959+0000","last_peered":"2024-04-26T02:35:19.473959+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:19.473959+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3281420,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11719,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1578}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11456","reported_seq":15671,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:18.553153+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:18.553153+0000","last_peered":"2024-04-26T02:35:18.553153+0000","last_clean":"2024-04-26T02:35:18.553153+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:18.553153+0000","last_undegraded":"2024-04-26T02:35:18.553153+0000","last_fullsized":"2024-04-26T02:35:18.553153+0000","mapping_epoch":35,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3265036,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11416","reported_seq":11651,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.027510+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:21.027510+0000","last_peered":"2024-04-26T02:35:21.027510+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:21.027510+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3117580,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":49675456,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181486,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10375,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99139584,"data_stored":99139584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679156,"kb_used_data":25516,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93692684,"statfs":{"total":96636764160,"available":95941308416,"internally_reserved":0,"allocated":26128384,"data_stored":25915341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,8,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667804,"kb_used_data":22988,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93704036,"statfs":{"total":96636764160,"available":95952932864,"internally_reserved":0,"allocated":23539712,"data_stored":23326669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019004,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559152,"kb_used_data":26344,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93812688,"statfs":{"total":96636764160,"available":96064192512,"internally_reserved":0,"allocated":26976256,"data_stored":26763213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669220,"kb_used_data":23964,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93702620,"statfs":{"total":96636764160,"available":95951482880,"internally_reserved":0,"allocated":24539136,"data_stored":24326093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,10,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24190976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26411008,"data_stored":26411008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23027712,"data_stored":23027712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25509888,"data_stored":25509888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:23.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:25.632 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:25.633 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:25.709 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-04-26T02:35:24.760023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":45169856,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182036,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10375,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2563452,"kb_used_data":91092,"kb_used_omap":14,"kb_used_meta":2472305,"kb_avail":374923908,"statfs":{"total":386547056640,"available":383922081792,"internally_reserved":0,"allocated":93278208,"data_stored":92426036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2531640883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,8,12,16,8,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20455424,"num_objects":-2497,"num_object_clones":0,"num_object_copies":-9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010204"},"pg_stats":[{"pgid":"1.f","version":"40'11187","reported_seq":11424,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.606754+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:24.606754+0000","last_peered":"2024-04-26T02:35:24.606754+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:24.606754+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11467","reported_seq":14308,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.215870+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:21.215870+0000","last_peered":"2024-04-26T02:35:21.215870+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:21.215870+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2986508,"num_objects":365,"num_object_clones":0,"num_object_copies":1460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11258","reported_seq":11433,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.397883+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:22.397883+0000","last_peered":"2024-04-26T02:35:22.397883+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:22.397883+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3045662,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11257,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1611}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11130","reported_seq":15161,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:23.633342+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:23.633342+0000","last_peered":"2024-04-26T02:35:23.633342+0000","last_clean":"2024-04-26T02:35:23.633342+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:23.633342+0000","last_undegraded":"2024-04-26T02:35:23.633342+0000","last_fullsized":"2024-04-26T02:35:23.633342+0000","mapping_epoch":37,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:24.434767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2863628,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11276","reported_seq":11512,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.377288+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:24.377288+0000","last_peered":"2024-04-26T02:35:24.377288+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:24.377288+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2626060,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11274,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11265","reported_seq":11526,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.356139+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:24.356139+0000","last_peered":"2024-04-26T02:35:24.356139+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:24.356139+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2744320,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11395","reported_seq":15653,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:21.052186+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:21.052186+0000","last_peered":"2024-04-26T02:35:21.052186+0000","last_clean":"2024-04-26T02:35:21.052186+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:21.052186+0000","last_undegraded":"2024-04-26T02:35:21.052186+0000","last_fullsized":"2024-04-26T02:35:21.052186+0000","mapping_epoch":33,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:22.166215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2970124,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11649","reported_seq":11909,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.164009+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:21.164009+0000","last_peered":"2024-04-26T02:35:21.164009+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:21.164009+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2781708,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11402","reported_seq":15496,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:24.618185+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:24.618185+0000","last_peered":"2024-04-26T02:35:24.618185+0000","last_clean":"2024-04-26T02:35:24.618185+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:24.618185+0000","last_undegraded":"2024-04-26T02:35:24.618185+0000","last_fullsized":"2024-04-26T02:35:24.618185+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:26.195780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2761752,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11248","reported_seq":11425,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.672884+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:22.672884+0000","last_peered":"2024-04-26T02:35:22.672884+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:22.672884+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585100,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11384","reported_seq":11624,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.586910+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:24.586910+0000","last_peered":"2024-04-26T02:35:24.586910+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:24.586910+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2738962,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11285","reported_seq":11520,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.622221+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:24.622221+0000","last_peered":"2024-04-26T02:35:24.622221+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:24.622221+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554118,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11284,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11411","reported_seq":11634,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:22.671981+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:22.671981+0000","last_peered":"2024-04-26T02:35:22.671981+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:22.671981+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3012870,"num_objects":368,"num_object_clones":0,"num_object_copies":1472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11773","reported_seq":12026,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.563830+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:24.563830+0000","last_peered":"2024-04-26T02:35:24.563830+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:24.563830+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855436,"num_objects":349,"num_object_clones":0,"num_object_copies":1396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11517","reported_seq":15732,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:23.717709+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:23.717709+0000","last_peered":"2024-04-26T02:35:23.717709+0000","last_clean":"2024-04-26T02:35:23.717709+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:23.717709+0000","last_undegraded":"2024-04-26T02:35:23.717709+0000","last_fullsized":"2024-04-26T02:35:23.717709+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:24.434743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2765324,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11416","reported_seq":11651,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:21.027510+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:21.027510+0000","last_peered":"2024-04-26T02:35:21.027510+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:21.027510+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3117580,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":45169856,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182036,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10375,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":91156480,"data_stored":91156480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662796,"kb_used_data":21636,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93709044,"statfs":{"total":96636764160,"available":95958061056,"internally_reserved":0,"allocated":22155264,"data_stored":21942221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667804,"kb_used_data":22988,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93704036,"statfs":{"total":96636764160,"available":95952932864,"internally_reserved":0,"allocated":23539712,"data_stored":23326669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563632,"kb_used_data":22504,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93808208,"statfs":{"total":96636764160,"available":96059604992,"internally_reserved":0,"allocated":23044096,"data_stored":22831053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,4,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669220,"kb_used_data":23964,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93702620,"statfs":{"total":96636764160,"available":95951482880,"internally_reserved":0,"allocated":24539136,"data_stored":24326093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660729551},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,10,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24190976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22421504,"data_stored":22421504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23027712,"data_stored":23027712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21516288,"data_stored":21516288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:25.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11187", 'reported_seq': 11424, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.606754+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:24.606754+0000', 'last_peered': '2024-04-26T02:35:24.606754+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:24.606754+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1100", 'ondisk_log_start': "28'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2760704, 'num_objects': 337, 'num_object_clones': 0, 'num_object_copies': 1348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11187, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1590}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11467", 'reported_seq': 14308, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:21.215870+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:21.215870+0000', 'last_peered': '2024-04-26T02:35:21.215870+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:21.215870+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2986508, 'num_objects': 365, 'num_object_clones': 0, 'num_object_copies': 1460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 698, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11465, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1382, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11258", 'reported_seq': 11433, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:22.397883+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:22.397883+0000', 'last_peered': '2024-04-26T02:35:22.397883+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:22.397883+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3045662, 'num_objects': 373, 'num_object_clones': 0, 'num_object_copies': 1492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 373, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11257, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1611}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11276", 'reported_seq': 11512, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.377288+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:24.377288+0000', 'last_peered': '2024-04-26T02:35:24.377288+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:24.377288+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2626060, 'num_objects': 321, 'num_object_clones': 0, 'num_object_copies': 1284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11274, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:25.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11265", 'reported_seq': 11526, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.356139+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:24.356139+0000', 'last_peered': '2024-04-26T02:35:24.356139+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:24.356139+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2744320, 'num_objects': 335, 'num_object_clones': 0, 'num_object_copies': 1340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11265, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1615}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11649", 'reported_seq': 11909, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:21.164009+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:21.164009+0000', 'last_peered': '2024-04-26T02:35:21.164009+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:21.164009+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2781708, 'num_objects': 340, 'num_object_clones': 0, 'num_object_copies': 1360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11647, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11248", 'reported_seq': 11425, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:22.672884+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:22.672884+0000', 'last_peered': '2024-04-26T02:35:22.672884+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:22.672884+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2585100, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11246, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1679}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11384", 'reported_seq': 11624, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.586910+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:24.586910+0000', 'last_peered': '2024-04-26T02:35:24.586910+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:24.586910+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2738962, 'num_objects': 335, 'num_object_clones': 0, 'num_object_copies': 1340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11381, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11285", 'reported_seq': 11520, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.622221+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:24.622221+0000', 'last_peered': '2024-04-26T02:35:24.622221+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:24.622221+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2554118, 'num_objects': 312, 'num_object_clones': 0, 'num_object_copies': 1248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11284, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:25.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11411", 'reported_seq': 11634, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:22.671981+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:22.671981+0000', 'last_peered': '2024-04-26T02:35:22.671981+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:22.671981+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3012870, 'num_objects': 368, 'num_object_clones': 0, 'num_object_copies': 1472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11410, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:25.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:25.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11773", 'reported_seq': 12026, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:24.563830+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:24.563830+0000', 'last_peered': '2024-04-26T02:35:24.563830+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:24.563830+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2855436, 'num_objects': 349, 'num_object_clones': 0, 'num_object_copies': 1396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11771, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:25.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:25.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11416", 'reported_seq': 11651, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:21.027510+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:21.027510+0000', 'last_peered': '2024-04-26T02:35:21.027510+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:21.027510+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3117580, 'num_objects': 381, 'num_object_clones': 0, 'num_object_copies': 1524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11414, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:25.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:28.225 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:28.225 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22328/21180 objects degraded (105.420%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":724,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5295,"data_bytes":43318464,"bytes_used":2643869696,"bytes_avail":383903186944,"bytes_total":386547056640,"degraded_objects":22328,"degraded_total":21180,"degraded_ratio":1.0542020774315393,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-26T02:34:30.728117+0000","services":{}},"progress_events":{}} 2024-04-26T02:35:28.338 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:28.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:30.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:30.112 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:30.208 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-04-26T02:35:28.760995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41721024,"num_objects":5100,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182457,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10441,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2600312,"kb_used_data":83544,"kb_used_omap":14,"kb_used_meta":2516721,"kb_avail":374887048,"statfs":{"total":386547056640,"available":383884337152,"internally_reserved":0,"allocated":85549056,"data_stored":84696884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2577122867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,11,8,9,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17408000,"num_objects":-2125,"num_object_clones":0,"num_object_copies":-8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006447"},"pg_stats":[{"pgid":"1.f","version":"40'11187","reported_seq":11424,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.606754+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:24.606754+0000","last_peered":"2024-04-26T02:35:24.606754+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:24.606754+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1590}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11525","reported_seq":14498,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:26.299460+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:26.299460+0000","last_peered":"2024-04-26T02:35:26.299460+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:26.299460+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2511372,"num_objects":307,"num_object_clones":0,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11320","reported_seq":11495,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.629483+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:27.629483+0000","last_peered":"2024-04-26T02:35:27.629483+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:27.629483+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2537758,"num_objects":311,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11319,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1673}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11130","reported_seq":15161,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:23.633342+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:23.633342+0000","last_peered":"2024-04-26T02:35:23.633342+0000","last_clean":"2024-04-26T02:35:23.633342+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:23.633342+0000","last_undegraded":"2024-04-26T02:35:23.633342+0000","last_fullsized":"2024-04-26T02:35:23.633342+0000","mapping_epoch":37,"log_start":"28'1100","ondisk_log_start":"28'1100","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:24.434767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2863628,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11276","reported_seq":11512,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.377288+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:24.377288+0000","last_peered":"2024-04-26T02:35:24.377288+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:24.377288+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2626060,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11274,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11265","reported_seq":11526,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.356139+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:24.356139+0000","last_peered":"2024-04-26T02:35:24.356139+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:24.356139+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2744320,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11451","reported_seq":15708,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:26.138882+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:26.138882+0000","last_peered":"2024-04-26T02:35:26.138882+0000","last_clean":"2024-04-26T02:35:26.138882+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:26.138882+0000","last_undegraded":"2024-04-26T02:35:26.138882+0000","last_fullsized":"2024-04-26T02:35:26.138882+0000","mapping_epoch":33,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:27.175874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2511372,"num_objects":307,"num_object_clones":0,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11711","reported_seq":11971,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:26.042053+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:26.042053+0000","last_peered":"2024-04-26T02:35:26.042053+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:26.042053+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2273804,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11709,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11402","reported_seq":15496,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:24.618185+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:24.618185+0000","last_peered":"2024-04-26T02:35:24.618185+0000","last_clean":"2024-04-26T02:35:24.618185+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:24.618185+0000","last_undegraded":"2024-04-26T02:35:24.618185+0000","last_fullsized":"2024-04-26T02:35:24.618185+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:26.195780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2761752,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11309","reported_seq":11488,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.754637+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:27.754637+0000","last_peered":"2024-04-26T02:35:27.754637+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:27.754637+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2085388,"num_objects":255,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11384","reported_seq":11624,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.586910+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:24.586910+0000","last_peered":"2024-04-26T02:35:24.586910+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:24.586910+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2738962,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11285","reported_seq":11520,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.622221+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:24.622221+0000","last_peered":"2024-04-26T02:35:24.622221+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:24.622221+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554118,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11284,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1586}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11483","reported_seq":11706,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.698697+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:27.698697+0000","last_peered":"2024-04-26T02:35:27.698697+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:27.698697+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2423046,"num_objects":296,"num_object_clones":0,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11773","reported_seq":12026,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:24.563830+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:24.563830+0000","last_peered":"2024-04-26T02:35:24.563830+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:24.563830+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855436,"num_objects":349,"num_object_clones":0,"num_object_copies":1396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11517","reported_seq":15732,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:23.717709+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:23.717709+0000","last_peered":"2024-04-26T02:35:23.717709+0000","last_clean":"2024-04-26T02:35:23.717709+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:23.717709+0000","last_undegraded":"2024-04-26T02:35:23.717709+0000","last_fullsized":"2024-04-26T02:35:23.717709+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:24.434743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2765324,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11466","reported_seq":11701,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:26.169091+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:26.169091+0000","last_peered":"2024-04-26T02:35:26.169091+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:26.169091+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2707980,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11464,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1656}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":41721024,"num_objects":5100,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182457,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10441,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83193856,"data_stored":83193856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662796,"kb_used_data":21636,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93709044,"statfs":{"total":96636764160,"available":95958061056,"internally_reserved":0,"allocated":22155264,"data_stored":21942221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686212,"kb_used_data":19196,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93685628,"statfs":{"total":96636764160,"available":95934083072,"internally_reserved":0,"allocated":19656704,"data_stored":19443661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019005,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563632,"kb_used_data":22504,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93808208,"statfs":{"total":96636764160,"available":96059604992,"internally_reserved":0,"allocated":23044096,"data_stored":22831053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,4,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687672,"kb_used_data":20208,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93684168,"statfs":{"total":96636764160,"available":95932588032,"internally_reserved":0,"allocated":20692992,"data_stored":20479949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":683470543},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,1,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20262912,"data_stored":20262912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22421504,"data_stored":22421504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18993152,"data_stored":18993152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21516288,"data_stored":21516288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:33.210 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:34.501 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:34.503 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:34.575 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-26T02:35:32.762064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":35347648,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183235,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10491,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2619392,"kb_used_data":72096,"kb_used_omap":14,"kb_used_meta":2547249,"kb_avail":374867968,"statfs":{"total":386547056640,"available":383864799232,"internally_reserved":0,"allocated":73826304,"data_stored":72974132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2608383539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,19,2,11,9,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17866752,"num_objects":-2181,"num_object_clones":0,"num_object_copies":-8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003054"},"pg_stats":[{"pgid":"1.f","version":"40'11236","reported_seq":11473,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.595960+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:29.595960+0000","last_peered":"2024-04-26T02:35:29.595960+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:29.595960+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11590","reported_seq":14664,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.427822+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:31.427822+0000","last_peered":"2024-04-26T02:35:31.427822+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:31.427822+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1978892,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11320","reported_seq":11495,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.629483+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:27.629483+0000","last_peered":"2024-04-26T02:35:27.629483+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:27.629483+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2537758,"num_objects":311,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11319,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1673}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11201","reported_seq":15232,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:28.670109+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:28.670109+0000","last_peered":"2024-04-26T02:35:28.670109+0000","last_clean":"2024-04-26T02:35:28.670109+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:28.670109+0000","last_undegraded":"2024-04-26T02:35:28.670109+0000","last_fullsized":"2024-04-26T02:35:28.670109+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:30.361193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2281996,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11342","reported_seq":11578,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.278052+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:29.278052+0000","last_peered":"2024-04-26T02:35:29.278052+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:29.278052+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2085388,"num_objects":255,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11340,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11322","reported_seq":11584,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.552533+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:29.552533+0000","last_peered":"2024-04-26T02:35:29.552533+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:29.552533+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2277376,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11322,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11521","reported_seq":15779,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:31.307919+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:31.307919+0000","last_peered":"2024-04-26T02:35:31.307919+0000","last_clean":"2024-04-26T02:35:31.307919+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:31.307919+0000","last_undegraded":"2024-04-26T02:35:31.307919+0000","last_fullsized":"2024-04-26T02:35:31.307919+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:32.242244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1937932,"num_objects":237,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11760","reported_seq":12020,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.118890+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:31.118890+0000","last_peered":"2024-04-26T02:35:31.118890+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:31.118890+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1872396,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11758,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1823}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11451","reported_seq":15547,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:29.501421+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:29.501421+0000","last_peered":"2024-04-26T02:35:29.501421+0000","last_clean":"2024-04-26T02:35:29.501421+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:29.501421+0000","last_undegraded":"2024-04-26T02:35:29.501421+0000","last_fullsized":"2024-04-26T02:35:29.501421+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:31.205971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2360344,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11309","reported_seq":11488,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.754637+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:27.754637+0000","last_peered":"2024-04-26T02:35:27.754637+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:27.754637+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2085388,"num_objects":255,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11446","reported_seq":11686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.677232+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:29.677232+0000","last_peered":"2024-04-26T02:35:29.677232+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:29.677232+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2231058,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11347","reported_seq":11582,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.596870+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:29.596870+0000","last_peered":"2024-04-26T02:35:29.596870+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:29.596870+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2046214,"num_objects":250,"num_object_clones":0,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1648}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11483","reported_seq":11706,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:27.698697+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:27.698697+0000","last_peered":"2024-04-26T02:35:27.698697+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:27.698697+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2423046,"num_objects":296,"num_object_clones":0,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1749}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11839","reported_seq":12090,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.588532+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:29.588532+0000","last_peered":"2024-04-26T02:35:29.588532+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:29.588532+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2314764,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11573","reported_seq":15789,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:28.842166+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:28.842166+0000","last_peered":"2024-04-26T02:35:28.842166+0000","last_clean":"2024-04-26T02:35:28.842166+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:28.842166+0000","last_undegraded":"2024-04-26T02:35:28.842166+0000","last_fullsized":"2024-04-26T02:35:28.842166+0000","mapping_epoch":35,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:30.361168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2306572,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11522","reported_seq":11757,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.368082+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:31.368082+0000","last_peered":"2024-04-26T02:35:31.368082+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:31.368082+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2249228,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":35347648,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183235,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10491,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":71204864,"data_stored":71204864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681264,"kb_used_data":17896,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93690576,"statfs":{"total":96636764160,"available":95939149824,"internally_reserved":0,"allocated":18325504,"data_stored":18112461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686212,"kb_used_data":19196,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93685628,"statfs":{"total":96636764160,"available":95934083072,"internally_reserved":0,"allocated":19656704,"data_stored":19443661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563976,"kb_used_data":18688,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93807864,"statfs":{"total":96636764160,"available":96059252736,"internally_reserved":0,"allocated":19136512,"data_stored":18923469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,9,1,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687940,"kb_used_data":16316,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93683900,"statfs":{"total":96636764160,"available":95932313600,"internally_reserved":0,"allocated":16707584,"data_stored":16494541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16187392,"data_stored":16187392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18419712,"data_stored":18419712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18993152,"data_stored":18993152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17604608,"data_stored":17604608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:34.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:36.070 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:36.071 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:36.176 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-04-26T02:35:34.762586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32857280,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183539,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10491,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2603240,"kb_used_data":64256,"kb_used_omap":14,"kb_used_meta":2538929,"kb_avail":374884120,"statfs":{"total":386547056640,"available":383881338880,"internally_reserved":0,"allocated":65798144,"data_stored":64945972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2599863859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,12,3,1,5,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16818176,"num_objects":-2053,"num_object_clones":0,"num_object_copies":-8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"1.f","version":"40'11236","reported_seq":11473,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.595960+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:29.595960+0000","last_peered":"2024-04-26T02:35:29.595960+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:29.595960+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11590","reported_seq":14664,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.427822+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:31.427822+0000","last_peered":"2024-04-26T02:35:31.427822+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:31.427822+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1978892,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11386","reported_seq":11561,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:33.046602+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:33.046602+0000","last_peered":"2024-04-26T02:35:33.046602+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:33.046602+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997086,"num_objects":245,"num_object_clones":0,"num_object_copies":980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11385,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11269","reported_seq":15300,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:33.703587+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:33.703587+0000","last_peered":"2024-04-26T02:35:33.703587+0000","last_clean":"2024-04-26T02:35:33.703587+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:33.703587+0000","last_undegraded":"2024-04-26T02:35:33.703587+0000","last_fullsized":"2024-04-26T02:35:33.703587+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:35.422897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724940,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11342","reported_seq":11578,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.278052+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:29.278052+0000","last_peered":"2024-04-26T02:35:29.278052+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:29.278052+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2085388,"num_objects":255,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11340,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11322","reported_seq":11584,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.552533+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:29.552533+0000","last_peered":"2024-04-26T02:35:29.552533+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:29.552533+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2277376,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11322,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11521","reported_seq":15779,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:31.307919+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:31.307919+0000","last_peered":"2024-04-26T02:35:31.307919+0000","last_clean":"2024-04-26T02:35:31.307919+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:31.307919+0000","last_undegraded":"2024-04-26T02:35:31.307919+0000","last_fullsized":"2024-04-26T02:35:31.307919+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:32.242244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1937932,"num_objects":237,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11760","reported_seq":12020,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.118890+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:31.118890+0000","last_peered":"2024-04-26T02:35:31.118890+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:31.118890+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1872396,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11758,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1823}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11451","reported_seq":15547,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:29.501421+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:29.501421+0000","last_peered":"2024-04-26T02:35:29.501421+0000","last_clean":"2024-04-26T02:35:29.501421+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:29.501421+0000","last_undegraded":"2024-04-26T02:35:29.501421+0000","last_fullsized":"2024-04-26T02:35:29.501421+0000","mapping_epoch":39,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:31.205971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2360344,"num_objects":289,"num_object_clones":0,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11356","reported_seq":11534,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:33.124468+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:33.124468+0000","last_peered":"2024-04-26T02:35:33.124468+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:33.124468+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1700364,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11446","reported_seq":11686,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.677232+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:29.677232+0000","last_peered":"2024-04-26T02:35:29.677232+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:29.677232+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2231058,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11347","reported_seq":11582,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.596870+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:29.596870+0000","last_peered":"2024-04-26T02:35:29.596870+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:29.596870+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2046214,"num_objects":250,"num_object_clones":0,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1648}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11543","reported_seq":11766,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:32.465610+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:32.465610+0000","last_peered":"2024-04-26T02:35:32.465610+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:32.465610+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931526,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11542,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1809}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11839","reported_seq":12090,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:29.588532+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:29.588532+0000","last_peered":"2024-04-26T02:35:29.588532+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:29.588532+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2314764,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11636","reported_seq":15852,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:33.876123+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:33.876123+0000","last_peered":"2024-04-26T02:35:33.876123+0000","last_clean":"2024-04-26T02:35:33.876123+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:33.876123+0000","last_undegraded":"2024-04-26T02:35:33.876123+0000","last_fullsized":"2024-04-26T02:35:33.876123+0000","mapping_epoch":35,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:35.422864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1790476,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11634,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11522","reported_seq":11757,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:31.368082+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:31.368082+0000","last_peered":"2024-04-26T02:35:31.368082+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:31.368082+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2249228,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32857280,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183539,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10491,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63029248,"data_stored":63029248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681264,"kb_used_data":17896,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93690576,"statfs":{"total":96636764160,"available":95939149824,"internally_reserved":0,"allocated":18325504,"data_stored":18112461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686440,"kb_used_data":15264,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93685400,"statfs":{"total":96636764160,"available":95933849600,"internally_reserved":0,"allocated":15630336,"data_stored":15417293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547596,"kb_used_data":14780,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93824244,"statfs":{"total":96636764160,"available":96076025856,"internally_reserved":0,"allocated":15134720,"data_stored":14921677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687940,"kb_used_data":16316,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93683900,"statfs":{"total":96636764160,"available":95932313600,"internally_reserved":0,"allocated":16707584,"data_stored":16494541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":687730383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16187392,"data_stored":16187392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14409728,"data_stored":14409728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14827520,"data_stored":14827520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17604608,"data_stored":17604608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11236", 'reported_seq': 11473, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.595960+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:29.595960+0000', 'last_peered': '2024-04-26T02:35:29.595960+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:29.595960+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1200", 'ondisk_log_start': "28'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2359296, 'num_objects': 288, 'num_object_clones': 0, 'num_object_copies': 1152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11236, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11590", 'reported_seq': 14664, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:31.427822+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:31.427822+0000', 'last_peered': '2024-04-26T02:35:31.427822+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:31.427822+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1978892, 'num_objects': 242, 'num_object_clones': 0, 'num_object_copies': 968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11588, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1498, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:36.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11386", 'reported_seq': 11561, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:33.046602+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:33.046602+0000', 'last_peered': '2024-04-26T02:35:33.046602+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:33.046602+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1997086, 'num_objects': 245, 'num_object_clones': 0, 'num_object_copies': 980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 245, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11385, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1739}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11342", 'reported_seq': 11578, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.278052+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:29.278052+0000', 'last_peered': '2024-04-26T02:35:29.278052+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:29.278052+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2085388, 'num_objects': 255, 'num_object_clones': 0, 'num_object_copies': 1020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11340, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1674}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11322", 'reported_seq': 11584, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.552533+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:29.552533+0000', 'last_peered': '2024-04-26T02:35:29.552533+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:29.552533+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2277376, 'num_objects': 278, 'num_object_clones': 0, 'num_object_copies': 1112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11322, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11760", 'reported_seq': 12020, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:31.118890+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:31.118890+0000', 'last_peered': '2024-04-26T02:35:31.118890+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:31.118890+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1872396, 'num_objects': 229, 'num_object_clones': 0, 'num_object_copies': 916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11758, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:36.178 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11356", 'reported_seq': 11534, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:33.124468+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:33.124468+0000', 'last_peered': '2024-04-26T02:35:33.124468+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:33.124468+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1700364, 'num_objects': 208, 'num_object_clones': 0, 'num_object_copies': 832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11354, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1787}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11446", 'reported_seq': 11686, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.677232+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:29.677232+0000', 'last_peered': '2024-04-26T02:35:29.677232+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:29.677232+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2231058, 'num_objects': 273, 'num_object_clones': 0, 'num_object_copies': 1092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11443, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1661}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11347", 'reported_seq': 11582, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.596870+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:29.596870+0000', 'last_peered': '2024-04-26T02:35:29.596870+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:29.596870+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2046214, 'num_objects': 250, 'num_object_clones': 0, 'num_object_copies': 1000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11543", 'reported_seq': 11766, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:32.465610+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:32.465610+0000', 'last_peered': '2024-04-26T02:35:32.465610+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:32.465610+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1931526, 'num_objects': 236, 'num_object_clones': 0, 'num_object_copies': 944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11542, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11839", 'reported_seq': 12090, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:29.588532+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:29.588532+0000', 'last_peered': '2024-04-26T02:35:29.588532+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:29.588532+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2314764, 'num_objects': 283, 'num_object_clones': 0, 'num_object_copies': 1132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11837, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:36.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11522", 'reported_seq': 11757, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:31.368082+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:31.368082+0000', 'last_peered': '2024-04-26T02:35:31.368082+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:31.368082+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2249228, 'num_objects': 275, 'num_object_clones': 0, 'num_object_copies': 1100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11520, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:36.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:38.034 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:38.034 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22225/13324 objects degraded (166.804%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":734,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3331,"data_bytes":27229376,"bytes_used":2649444352,"bytes_avail":383897612288,"bytes_total":386547056640,"degraded_objects":22225,"degraded_total":13324,"degraded_ratio":1.6680426298408886,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":182},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:35:38.137 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:38.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:39.740 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:39.741 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:39.886 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-04-26T02:35:38.763638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25910464,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10544,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160614,"ondisk_log_size":160614,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2589172,"kb_used_data":52928,"kb_used_omap":14,"kb_used_meta":2536177,"kb_avail":374898188,"statfs":{"total":386547056640,"available":383895744512,"internally_reserved":0,"allocated":54198272,"data_stored":53346100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2597045811},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,11,22,11,10,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17408000,"num_objects":-2125,"num_object_clones":0,"num_object_copies":-8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003120"},"pg_stats":[{"pgid":"1.f","version":"40'11308","reported_seq":11545,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.574865+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:34.574865+0000","last_peered":"2024-04-26T02:35:34.574865+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:34.574865+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11308,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11652","reported_seq":14833,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:36.465871+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:36.465871+0000","last_peered":"2024-04-26T02:35:36.465871+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:36.465871+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1470988,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1551,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11446","reported_seq":11620,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.199124+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:38.199124+0000","last_peered":"2024-04-26T02:35:38.199124+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:38.199124+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1505566,"num_objects":185,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11445,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11269","reported_seq":15300,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:33.703587+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:33.703587+0000","last_peered":"2024-04-26T02:35:33.703587+0000","last_clean":"2024-04-26T02:35:33.703587+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:33.703587+0000","last_undegraded":"2024-04-26T02:35:33.703587+0000","last_fullsized":"2024-04-26T02:35:33.703587+0000","mapping_epoch":37,"log_start":"28'1200","ondisk_log_start":"28'1200","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:35.422897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724940,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11405","reported_seq":11640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.765018+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:34.765018+0000","last_peered":"2024-04-26T02:35:34.765018+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:34.765018+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1569292,"num_objects":192,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11376","reported_seq":11638,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.693035+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:34.693035+0000","last_peered":"2024-04-26T02:35:34.693035+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:34.693035+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1835008,"num_objects":224,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11581","reported_seq":15838,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:36.518560+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:36.518560+0000","last_peered":"2024-04-26T02:35:36.518560+0000","last_clean":"2024-04-26T02:35:36.518560+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:36.518560+0000","last_undegraded":"2024-04-26T02:35:36.518560+0000","last_fullsized":"2024-04-26T02:35:36.518560+0000","mapping_epoch":33,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:38.199741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1446412,"num_objects":177,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11814","reported_seq":12074,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:35.831090+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:35.831090+0000","last_peered":"2024-04-26T02:35:35.831090+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:35.831090+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1430028,"num_objects":175,"num_object_clones":0,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1877}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11521","reported_seq":15616,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:34.695559+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:34.695559+0000","last_peered":"2024-04-26T02:35:34.695559+0000","last_clean":"2024-04-26T02:35:34.695559+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:34.695559+0000","last_undegraded":"2024-04-26T02:35:34.695559+0000","last_fullsized":"2024-04-26T02:35:34.695559+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:37.158634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1786904,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11405","reported_seq":11584,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.140977+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:38.140977+0000","last_peered":"2024-04-26T02:35:38.140977+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:38.140977+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1298956,"num_objects":159,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11508","reported_seq":11748,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.634651+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:34.634651+0000","last_peered":"2024-04-26T02:35:34.634651+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:34.634651+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1723154,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11405","reported_seq":11638,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.677044+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:34.677044+0000","last_peered":"2024-04-26T02:35:34.677044+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:34.677044+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1571078,"num_objects":192,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1706}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11595","reported_seq":11818,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.240202+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:38.240202+0000","last_peered":"2024-04-26T02:35:38.240202+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:38.240202+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1505542,"num_objects":184,"num_object_clones":0,"num_object_copies":736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11904","reported_seq":12157,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:34.690835+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:34.690835+0000","last_peered":"2024-04-26T02:35:34.690835+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:34.690835+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1782284,"num_objects":218,"num_object_clones":0,"num_object_copies":872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11636","reported_seq":15852,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:33.876123+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:33.876123+0000","last_peered":"2024-04-26T02:35:33.876123+0000","last_clean":"2024-04-26T02:35:33.876123+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:33.876123+0000","last_undegraded":"2024-04-26T02:35:33.876123+0000","last_fullsized":"2024-04-26T02:35:33.876123+0000","mapping_epoch":35,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:35.422864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1790476,"num_objects":219,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11634,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11589","reported_seq":11824,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:36.338256+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:36.338256+0000","last_peered":"2024-04-26T02:35:36.338256+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:36.338256+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1700364,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":25910464,"num_objects":3170,"num_object_clones":0,"num_object_copies":12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184387,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10544,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":51208192,"data_stored":51208192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160614,"ondisk_log_size":160614,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681632,"kb_used_data":14104,"kb_used_omap":4,"kb_used_meta":667515,"kb_avail":93690208,"statfs":{"total":96636764160,"available":95938772992,"internally_reserved":0,"allocated":14442496,"data_stored":14229453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":683536080},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,10,5,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688264,"kb_used_data":11520,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93683576,"statfs":{"total":96636764160,"available":95931981824,"internally_reserved":0,"allocated":11796480,"data_stored":11583437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547596,"kb_used_data":14780,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93824244,"statfs":{"total":96636764160,"available":96076025856,"internally_reserved":0,"allocated":15134720,"data_stored":14921677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671680,"kb_used_data":12524,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93700160,"statfs":{"total":96636764160,"available":95948963840,"internally_reserved":0,"allocated":12824576,"data_stored":12611533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":674950863},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12292096,"data_stored":12292096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14409728,"data_stored":14409728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10907648,"data_stored":10907648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13598720,"data_stored":13598720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:42.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:44.590 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:44.591 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:44.715 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-04-26T02:35:42.764695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19528896,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185166,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160993,"ondisk_log_size":160993,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2626420,"kb_used_data":41620,"kb_used_omap":14,"kb_used_meta":2584753,"kb_avail":374860940,"statfs":{"total":386547056640,"available":383857602560,"internally_reserved":0,"allocated":42618880,"data_stored":41766708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2646787635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,4,14,8,8,3,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17784832,"num_objects":-2171,"num_object_clones":0,"num_object_copies":-8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"1.f","version":"40'11368","reported_seq":11606,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.962601+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:39.962601+0000","last_peered":"2024-04-26T02:35:39.962601+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:39.962601+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11699","reported_seq":15006,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.525466+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:41.525466+0000","last_peered":"2024-04-26T02:35:41.525466+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:41.525466+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1085964,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11446","reported_seq":11620,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.199124+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:38.199124+0000","last_peered":"2024-04-26T02:35:38.199124+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:38.199124+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1505566,"num_objects":185,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11445,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11328","reported_seq":15359,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:38.675615+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:38.675615+0000","last_peered":"2024-04-26T02:35:38.675615+0000","last_clean":"2024-04-26T02:35:38.675615+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:38.675615+0000","last_undegraded":"2024-04-26T02:35:38.675615+0000","last_fullsized":"2024-04-26T02:35:38.675615+0000","mapping_epoch":37,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:40.538197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1241612,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11457","reported_seq":11693,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.764400+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:39.764400+0000","last_peered":"2024-04-26T02:35:39.764400+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:39.764400+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1143308,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11455,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11448","reported_seq":11710,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.688100+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:39.688100+0000","last_peered":"2024-04-26T02:35:39.688100+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:39.688100+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11630","reported_seq":15888,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:41.538367+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:41.538367+0000","last_peered":"2024-04-26T02:35:41.538367+0000","last_clean":"2024-04-26T02:35:41.538367+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:41.538367+0000","last_undegraded":"2024-04-26T02:35:41.538367+0000","last_fullsized":"2024-04-26T02:35:41.538367+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.330401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1045004,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11628,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11876","reported_seq":12136,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.426901+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:41.426901+0000","last_peered":"2024-04-26T02:35:41.426901+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:41.426901+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":922124,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11874,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11579","reported_seq":15675,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:39.859925+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:39.859925+0000","last_peered":"2024-04-26T02:35:39.859925+0000","last_clean":"2024-04-26T02:35:39.859925+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:39.859925+0000","last_undegraded":"2024-04-26T02:35:39.859925+0000","last_fullsized":"2024-04-26T02:35:39.859925+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:42.185921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1311768,"num_objects":161,"num_object_clones":0,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11405","reported_seq":11584,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.140977+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:38.140977+0000","last_peered":"2024-04-26T02:35:38.140977+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:38.140977+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1298956,"num_objects":159,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1836}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11567","reported_seq":11807,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.890241+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:39.890241+0000","last_peered":"2024-04-26T02:35:39.890241+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:39.890241+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239826,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11469","reported_seq":11703,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.962134+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:39.962134+0000","last_peered":"2024-04-26T02:35:39.962134+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:39.962134+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1046790,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11468,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1770}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11595","reported_seq":11818,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:38.240202+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:38.240202+0000","last_peered":"2024-04-26T02:35:38.240202+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:38.240202+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1505542,"num_objects":184,"num_object_clones":0,"num_object_copies":736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11975","reported_seq":12228,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.652087+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:39.652087+0000","last_peered":"2024-04-26T02:35:39.652087+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:39.652087+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1200652,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11700","reported_seq":15916,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:39.036895+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:39.036895+0000","last_peered":"2024-04-26T02:35:39.036895+0000","last_clean":"2024-04-26T02:35:39.036895+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:39.036895+0000","last_undegraded":"2024-04-26T02:35:39.036895+0000","last_fullsized":"2024-04-26T02:35:39.036895+0000","mapping_epoch":35,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:40.515351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1266188,"num_objects":155,"num_object_clones":0,"num_object_copies":620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11651","reported_seq":11886,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.532504+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:41.532504+0000","last_peered":"2024-04-26T02:35:41.532504+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:41.532504+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1192460,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":19528896,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185166,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39424000,"data_stored":39424000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160993,"ondisk_log_size":160993,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700096,"kb_used_data":10360,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93671744,"statfs":{"total":96636764160,"available":95919865856,"internally_reserved":0,"allocated":10608640,"data_stored":10395597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,3,4,0,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688264,"kb_used_data":11520,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93683576,"statfs":{"total":96636764160,"available":95931981824,"internally_reserved":0,"allocated":11796480,"data_stored":11583437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548028,"kb_used_data":11060,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93823812,"statfs":{"total":96636764160,"available":96075583488,"internally_reserved":0,"allocated":11325440,"data_stored":11112397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690032,"kb_used_data":8680,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93681808,"statfs":{"total":96636764160,"available":95930171392,"internally_reserved":0,"allocated":8888320,"data_stored":8675277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":8351744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10522624,"data_stored":10522624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10907648,"data_stored":10907648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9641984,"data_stored":9641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:44.716 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:46.073 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:46.074 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:46.170 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-04-26T02:35:44.765189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16956608,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185480,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2627172,"kb_used_data":34028,"kb_used_omap":14,"kb_used_meta":2593073,"kb_avail":374860188,"statfs":{"total":386547056640,"available":383856832512,"internally_reserved":0,"allocated":34844672,"data_stored":33992500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2655307315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,1,11,14,10,9,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18391040,"num_objects":-2245,"num_object_clones":0,"num_object_copies":-8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2245,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.f","version":"40'11368","reported_seq":11606,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.962601+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:39.962601+0000","last_peered":"2024-04-26T02:35:39.962601+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:39.962601+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":156,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11699","reported_seq":15006,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.525466+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:41.525466+0000","last_peered":"2024-04-26T02:35:41.525466+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:41.525466+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1085964,"num_objects":133,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11516","reported_seq":11691,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:42.954807+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:42.954807+0000","last_peered":"2024-04-26T02:35:42.954807+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:42.954807+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932126,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11515,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1869}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11379","reported_seq":15410,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:43.904824+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:43.904824+0000","last_peered":"2024-04-26T02:35:43.904824+0000","last_clean":"2024-04-26T02:35:43.904824+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:43.904824+0000","last_undegraded":"2024-04-26T02:35:43.904824+0000","last_fullsized":"2024-04-26T02:35:43.904824+0000","mapping_epoch":37,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:46.418257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":823820,"num_objects":101,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11457","reported_seq":11693,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.764400+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:39.764400+0000","last_peered":"2024-04-26T02:35:39.764400+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:39.764400+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1143308,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11455,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11448","reported_seq":11710,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.688100+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:39.688100+0000","last_peered":"2024-04-26T02:35:39.688100+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:39.688100+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11630","reported_seq":15888,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:41.538367+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:41.538367+0000","last_peered":"2024-04-26T02:35:41.538367+0000","last_clean":"2024-04-26T02:35:41.538367+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:41.538367+0000","last_undegraded":"2024-04-26T02:35:41.538367+0000","last_fullsized":"2024-04-26T02:35:41.538367+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.330401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1045004,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11628,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11876","reported_seq":12136,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.426901+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:41.426901+0000","last_peered":"2024-04-26T02:35:41.426901+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:41.426901+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":922124,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11874,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11579","reported_seq":15675,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:39.859925+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:39.859925+0000","last_peered":"2024-04-26T02:35:39.859925+0000","last_clean":"2024-04-26T02:35:39.859925+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:39.859925+0000","last_undegraded":"2024-04-26T02:35:39.859925+0000","last_fullsized":"2024-04-26T02:35:39.859925+0000","mapping_epoch":39,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:42.185921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1311768,"num_objects":161,"num_object_clones":0,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11473","reported_seq":11652,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:43.415040+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:43.415040+0000","last_peered":"2024-04-26T02:35:43.415040+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:43.415040+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":741900,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11471,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11567","reported_seq":11807,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.890241+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:39.890241+0000","last_peered":"2024-04-26T02:35:39.890241+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:39.890241+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239826,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11469","reported_seq":11703,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.962134+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:39.962134+0000","last_peered":"2024-04-26T02:35:39.962134+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:39.962134+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1046790,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11468,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1770}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11661","reported_seq":11884,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:43.350984+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:43.350984+0000","last_peered":"2024-04-26T02:35:43.350984+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:43.350984+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964870,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1927}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'11975","reported_seq":12228,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:39.652087+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:39.652087+0000","last_peered":"2024-04-26T02:35:39.652087+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:39.652087+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1200652,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11759","reported_seq":15973,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:43.946677+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:43.946677+0000","last_peered":"2024-04-26T02:35:43.946677+0000","last_clean":"2024-04-26T02:35:43.946677+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:43.946677+0000","last_undegraded":"2024-04-26T02:35:43.946677+0000","last_fullsized":"2024-04-26T02:35:43.946677+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:46.418231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":782860,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11651","reported_seq":11886,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:41.532504+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:41.532504+0000","last_peered":"2024-04-26T02:35:41.532504+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:41.532504+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1192460,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16956608,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185480,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":31432704,"data_stored":31432704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700096,"kb_used_data":10360,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93671744,"statfs":{"total":96636764160,"available":95919865856,"internally_reserved":0,"allocated":10608640,"data_stored":10395597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,3,4,0,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688632,"kb_used_data":7704,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93683208,"statfs":{"total":96636764160,"available":95931604992,"internally_reserved":0,"allocated":7888896,"data_stored":7675853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548412,"kb_used_data":7284,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93823428,"statfs":{"total":96636764160,"available":96075190272,"internally_reserved":0,"allocated":7458816,"data_stored":7245773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690032,"kb_used_data":8680,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93681808,"statfs":{"total":96636764160,"available":95930171392,"internally_reserved":0,"allocated":8888320,"data_stored":8675277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697691855},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":8351744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6565888,"data_stored":6565888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6873088,"data_stored":6873088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9641984,"data_stored":9641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:46.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:46.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11368", 'reported_seq': 11606, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.962601+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:39.962601+0000', 'last_peered': '2024-04-26T02:35:39.962601+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:39.962601+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1300", 'ondisk_log_start': "28'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1277952, 'num_objects': 156, 'num_object_clones': 0, 'num_object_copies': 624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11368, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1771}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11699", 'reported_seq': 15006, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:41.525466+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:41.525466+0000', 'last_peered': '2024-04-26T02:35:41.525466+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:41.525466+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1085964, 'num_objects': 133, 'num_object_clones': 0, 'num_object_copies': 532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 466, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11697, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1614, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11516", 'reported_seq': 11691, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:42.954807+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:42.954807+0000', 'last_peered': '2024-04-26T02:35:42.954807+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:42.954807+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 932126, 'num_objects': 115, 'num_object_clones': 0, 'num_object_copies': 460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 115, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11515, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1869}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11457", 'reported_seq': 11693, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.764400+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:39.764400+0000', 'last_peered': '2024-04-26T02:35:39.764400+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:39.764400+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1143308, 'num_objects': 140, 'num_object_clones': 0, 'num_object_copies': 560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11455, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1789}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11448", 'reported_seq': 11710, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.688100+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:39.688100+0000', 'last_peered': '2024-04-26T02:35:39.688100+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:39.688100+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1245184, 'num_objects': 152, 'num_object_clones': 0, 'num_object_copies': 608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11448, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1798}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:46.172 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11876", 'reported_seq': 12136, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:41.426901+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:41.426901+0000', 'last_peered': '2024-04-26T02:35:41.426901+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:41.426901+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 922124, 'num_objects': 113, 'num_object_clones': 0, 'num_object_copies': 452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11874, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11473", 'reported_seq': 11652, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:43.415040+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:43.415040+0000', 'last_peered': '2024-04-26T02:35:43.415040+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:43.415040+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 741900, 'num_objects': 91, 'num_object_clones': 0, 'num_object_copies': 364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 91, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11471, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1904}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11567", 'reported_seq': 11807, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.890241+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:39.890241+0000', 'last_peered': '2024-04-26T02:35:39.890241+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:39.890241+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1239826, 'num_objects': 152, 'num_object_clones': 0, 'num_object_copies': 608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11564, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11469", 'reported_seq': 11703, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.962134+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:39.962134+0000', 'last_peered': '2024-04-26T02:35:39.962134+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:39.962134+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1046790, 'num_objects': 128, 'num_object_clones': 0, 'num_object_copies': 512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11468, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11661", 'reported_seq': 11884, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:43.350984+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:43.350984+0000', 'last_peered': '2024-04-26T02:35:43.350984+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:43.350984+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 964870, 'num_objects': 118, 'num_object_clones': 0, 'num_object_copies': 472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11660, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:46.173 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'11975", 'reported_seq': 12228, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:39.652087+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:39.652087+0000', 'last_peered': '2024-04-26T02:35:39.652087+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:39.652087+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1200652, 'num_objects': 147, 'num_object_clones': 0, 'num_object_copies': 588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11973, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:46.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:46.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11651", 'reported_seq': 11886, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:41.532504+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:41.532504+0000', 'last_peered': '2024-04-26T02:35:41.532504+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:41.532504+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1192460, 'num_objects': 146, 'num_object_clones': 0, 'num_object_copies': 584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11649, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:46.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:48.063 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22094/5608 objects degraded (393.973%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":744,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1402,"data_bytes":11427008,"bytes_used":2678046720,"bytes_avail":383869009920,"bytes_total":386547056640,"degraded_objects":22094,"degraded_total":5608,"degraded_ratio":3.9397289586305275,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":217},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:35:48.156 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:48.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:49.650 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:49.651 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:49.802 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-04-26T02:35:48.766191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10042560,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186324,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10675,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160651,"ondisk_log_size":160651,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2615720,"kb_used_data":22584,"kb_used_omap":14,"kb_used_meta":2593073,"kb_avail":374871640,"statfs":{"total":386547056640,"available":383868559360,"internally_reserved":0,"allocated":23126016,"data_stored":22273844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2655307315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,4,10,11,11,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17186816,"num_objects":-2098,"num_object_clones":0,"num_object_copies":-8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2098,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"1.f","version":"40'11433","reported_seq":11671,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:44.669229+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:44.669229+0000","last_peered":"2024-04-26T02:35:44.669229+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:44.669229+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":745472,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11433,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11759","reported_seq":15202,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:46.570525+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:46.570525+0000","last_peered":"2024-04-26T02:35:46.570525+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:46.570525+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":594444,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11581","reported_seq":11753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.434245+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:48.434245+0000","last_peered":"2024-04-26T02:35:48.434245+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:48.434245+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399646,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11580,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11379","reported_seq":15410,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:43.904824+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:43.904824+0000","last_peered":"2024-04-26T02:35:43.904824+0000","last_clean":"2024-04-26T02:35:43.904824+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:43.904824+0000","last_undegraded":"2024-04-26T02:35:43.904824+0000","last_fullsized":"2024-04-26T02:35:43.904824+0000","mapping_epoch":37,"log_start":"28'1300","ondisk_log_start":"28'1300","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:46.418257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":823820,"num_objects":101,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11525","reported_seq":11761,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:44.773865+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:44.773865+0000","last_peered":"2024-04-26T02:35:44.773865+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:44.773865+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586252,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11509","reported_seq":11771,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:44.735779+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:44.735779+0000","last_peered":"2024-04-26T02:35:44.735779+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:44.735779+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":745472,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1859}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11686","reported_seq":15944,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:46.244602+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:46.244602+0000","last_peered":"2024-04-26T02:35:46.244602+0000","last_clean":"2024-04-26T02:35:46.244602+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:46.244602+0000","last_undegraded":"2024-04-26T02:35:46.244602+0000","last_fullsized":"2024-04-26T02:35:46.244602+0000","mapping_epoch":33,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:48.413372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586252,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11684,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11937","reported_seq":12197,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:46.435237+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:46.435237+0000","last_peered":"2024-04-26T02:35:46.435237+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:46.435237+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422412,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11935,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2000}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11642","reported_seq":15738,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:44.975030+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:44.975030+0000","last_peered":"2024-04-26T02:35:44.975030+0000","last_clean":"2024-04-26T02:35:44.975030+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:44.975030+0000","last_undegraded":"2024-04-26T02:35:44.975030+0000","last_fullsized":"2024-04-26T02:35:44.975030+0000","mapping_epoch":39,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:47.238667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":795672,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11639,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11512","reported_seq":11691,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.288402+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:48.288402+0000","last_peered":"2024-04-26T02:35:48.288402+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:48.288402+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422412,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11632","reported_seq":11871,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:45.011120+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:45.011120+0000","last_peered":"2024-04-26T02:35:45.011120+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:45.011120+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":707346,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11523","reported_seq":11758,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:45.002760+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:45.002760+0000","last_peered":"2024-04-26T02:35:45.002760+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:45.002760+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604422,"num_objects":74,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1824}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11726","reported_seq":11949,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.369395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:48.369395+0000","last_peered":"2024-04-26T02:35:48.369395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:48.369395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432390,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12043","reported_seq":12294,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:44.803826+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:44.803826+0000","last_peered":"2024-04-26T02:35:44.803826+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:44.803826+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643596,"num_objects":79,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12041,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11759","reported_seq":15973,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:43.946677+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:43.946677+0000","last_peered":"2024-04-26T02:35:43.946677+0000","last_clean":"2024-04-26T02:35:43.946677+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:43.946677+0000","last_undegraded":"2024-04-26T02:35:43.946677+0000","last_fullsized":"2024-04-26T02:35:43.946677+0000","mapping_epoch":35,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:46.418231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":782860,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11705","reported_seq":11940,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:46.454974+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:46.454974+0000","last_peered":"2024-04-26T02:35:46.454974+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:46.454974+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":750092,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11703,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":10042560,"num_objects":1233,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186324,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10675,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19795968,"data_stored":19795968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160651,"ondisk_log_size":160651,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687880,"kb_used_data":6452,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93683960,"statfs":{"total":96636764160,"available":95932375040,"internally_reserved":0,"allocated":6606848,"data_stored":6393805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689072,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93682768,"statfs":{"total":96636764160,"available":95931154432,"internally_reserved":0,"allocated":4104192,"data_stored":3891149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548412,"kb_used_data":7284,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93823428,"statfs":{"total":96636764160,"available":96075190272,"internally_reserved":0,"allocated":7458816,"data_stored":7245773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690356,"kb_used_data":4840,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93681484,"statfs":{"total":96636764160,"available":95929839616,"internally_reserved":0,"allocated":4956160,"data_stored":4743117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":701951695},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4468736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6565888,"data_stored":6565888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3072000,"data_stored":3072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5689344,"data_stored":5689344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:52.290 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Removed 93736 objects 2024-04-26T02:35:52.291 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi177.stdout:Clean up completed and total clean up time :450.492 2024-04-26T02:35:52.804 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:53.105 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:53.106 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:53.121 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-04-26T02:35:52.771076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634932,"kb_used_data":11280,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374852428,"statfs":{"total":386547056640,"available":383848886272,"internally_reserved":0,"allocated":11550720,"data_stored":10698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,6,14,13,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17522688,"num_objects":-2139,"num_object_clones":0,"num_object_copies":-8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006868"},"pg_stats":[{"pgid":"1.f","version":"40'11494","reported_seq":11732,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.961050+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:49.961050+0000","last_peered":"2024-04-26T02:35:49.961050+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:49.961050+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760,"num_objects":30,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11581","reported_seq":11753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.434245+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:48.434245+0000","last_peered":"2024-04-26T02:35:48.434245+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:48.434245+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399646,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11580,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11445","reported_seq":15476,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.107744+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:49.107744+0000","last_peered":"2024-04-26T02:35:49.107744+0000","last_clean":"2024-04-26T02:35:49.107744+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:49.107744+0000","last_undegraded":"2024-04-26T02:35:49.107744+0000","last_fullsized":"2024-04-26T02:35:49.107744+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283148,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11578","reported_seq":11814,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.850991+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:49.850991+0000","last_peered":"2024-04-26T02:35:49.850991+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:49.850991+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152076,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11568","reported_seq":11830,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.754826+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:49.754826+0000","last_peered":"2024-04-26T02:35:49.754826+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:49.754826+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11568,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11751","reported_seq":16009,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.271862+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:51.271862+0000","last_peered":"2024-04-26T02:35:51.271862+0000","last_clean":"2024-04-26T02:35:51.271862+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:51.271862+0000","last_undegraded":"2024-04-26T02:35:51.271862+0000","last_fullsized":"2024-04-26T02:35:51.271862+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:54.380420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53772,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11984","reported_seq":12244,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.578568+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:51.578568+0000","last_peered":"2024-04-26T02:35:51.578568+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:51.578568+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11705","reported_seq":15801,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.749587+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:49.749587+0000","last_peered":"2024-04-26T02:35:49.749587+0000","last_clean":"2024-04-26T02:35:49.749587+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:49.749587+0000","last_undegraded":"2024-04-26T02:35:49.749587+0000","last_fullsized":"2024-04-26T02:35:49.749587+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.260886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11512","reported_seq":11691,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.288402+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:48.288402+0000","last_peered":"2024-04-26T02:35:48.288402+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:48.288402+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422412,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11691","reported_seq":11931,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.784695+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:49.784695+0000","last_peered":"2024-04-26T02:35:49.784695+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:49.784695+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224018,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11572","reported_seq":11807,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.960714+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:49.960714+0000","last_peered":"2024-04-26T02:35:49.960714+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:49.960714+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203014,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1873}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11726","reported_seq":11949,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.369395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:48.369395+0000","last_peered":"2024-04-26T02:35:48.369395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:48.369395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432390,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12088","reported_seq":12341,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.869615+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:49.869615+0000","last_peered":"2024-04-26T02:35:49.869615+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:49.869615+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274956,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11815","reported_seq":16031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.132122+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:49.132122+0000","last_peered":"2024-04-26T02:35:49.132122+0000","last_clean":"2024-04-26T02:35:49.132122+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:49.132122+0000","last_undegraded":"2024-04-26T02:35:49.132122+0000","last_fullsized":"2024-04-26T02:35:49.132122+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324108,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11784","reported_seq":12018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.629540+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:51.629540+0000","last_peered":"2024-04-26T02:35:51.629540+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:51.629540+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102924,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1974}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8134656,"data_stored":8134656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688276,"kb_used_data":2700,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683564,"statfs":{"total":96636764160,"available":95931969536,"internally_reserved":0,"allocated":2764800,"data_stored":2551757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689072,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93682768,"statfs":{"total":96636764160,"available":95931154432,"internally_reserved":0,"allocated":4104192,"data_stored":3891149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566872,"kb_used_data":3532,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93804968,"statfs":{"total":96636764160,"available":96056287232,"internally_reserved":0,"allocated":3616768,"data_stored":3403725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":576843471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690712,"kb_used_data":1040,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93681128,"statfs":{"total":96636764160,"available":95929475072,"internally_reserved":0,"allocated":1064960,"data_stored":851917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":2674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3072000,"data_stored":3072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1839104,"data_stored":1839104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:53.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:53.409 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:53.426 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-04-26T02:35:52.771076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634932,"kb_used_data":11280,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374852428,"statfs":{"total":386547056640,"available":383848886272,"internally_reserved":0,"allocated":11550720,"data_stored":10698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,6,14,13,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17522688,"num_objects":-2139,"num_object_clones":0,"num_object_copies":-8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006868"},"pg_stats":[{"pgid":"1.f","version":"40'11494","reported_seq":11732,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.961050+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:49.961050+0000","last_peered":"2024-04-26T02:35:49.961050+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:49.961050+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760,"num_objects":30,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11581","reported_seq":11753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.434245+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:48.434245+0000","last_peered":"2024-04-26T02:35:48.434245+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:48.434245+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399646,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11580,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11445","reported_seq":15476,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.107744+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:49.107744+0000","last_peered":"2024-04-26T02:35:49.107744+0000","last_clean":"2024-04-26T02:35:49.107744+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:49.107744+0000","last_undegraded":"2024-04-26T02:35:49.107744+0000","last_fullsized":"2024-04-26T02:35:49.107744+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283148,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11578","reported_seq":11814,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.850991+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:49.850991+0000","last_peered":"2024-04-26T02:35:49.850991+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:49.850991+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152076,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11568","reported_seq":11830,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.754826+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:49.754826+0000","last_peered":"2024-04-26T02:35:49.754826+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:49.754826+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11568,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11751","reported_seq":16009,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.271862+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:51.271862+0000","last_peered":"2024-04-26T02:35:51.271862+0000","last_clean":"2024-04-26T02:35:51.271862+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:51.271862+0000","last_undegraded":"2024-04-26T02:35:51.271862+0000","last_fullsized":"2024-04-26T02:35:51.271862+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:54.380420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53772,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11984","reported_seq":12244,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.578568+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:51.578568+0000","last_peered":"2024-04-26T02:35:51.578568+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:51.578568+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11705","reported_seq":15801,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.749587+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:49.749587+0000","last_peered":"2024-04-26T02:35:49.749587+0000","last_clean":"2024-04-26T02:35:49.749587+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:49.749587+0000","last_undegraded":"2024-04-26T02:35:49.749587+0000","last_fullsized":"2024-04-26T02:35:49.749587+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.260886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11512","reported_seq":11691,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.288402+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:48.288402+0000","last_peered":"2024-04-26T02:35:48.288402+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:48.288402+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422412,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11691","reported_seq":11931,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.784695+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:49.784695+0000","last_peered":"2024-04-26T02:35:49.784695+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:49.784695+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224018,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11572","reported_seq":11807,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.960714+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:49.960714+0000","last_peered":"2024-04-26T02:35:49.960714+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:49.960714+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203014,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1873}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11726","reported_seq":11949,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.369395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:48.369395+0000","last_peered":"2024-04-26T02:35:48.369395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:48.369395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432390,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12088","reported_seq":12341,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.869615+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:49.869615+0000","last_peered":"2024-04-26T02:35:49.869615+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:49.869615+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274956,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11815","reported_seq":16031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.132122+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:49.132122+0000","last_peered":"2024-04-26T02:35:49.132122+0000","last_clean":"2024-04-26T02:35:49.132122+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:49.132122+0000","last_undegraded":"2024-04-26T02:35:49.132122+0000","last_fullsized":"2024-04-26T02:35:49.132122+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324108,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11784","reported_seq":12018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.629540+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:51.629540+0000","last_peered":"2024-04-26T02:35:51.629540+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:51.629540+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102924,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1974}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8134656,"data_stored":8134656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688276,"kb_used_data":2700,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683564,"statfs":{"total":96636764160,"available":95931969536,"internally_reserved":0,"allocated":2764800,"data_stored":2551757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689072,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93682768,"statfs":{"total":96636764160,"available":95931154432,"internally_reserved":0,"allocated":4104192,"data_stored":3891149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566872,"kb_used_data":3532,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93804968,"statfs":{"total":96636764160,"available":96056287232,"internally_reserved":0,"allocated":3616768,"data_stored":3403725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":576843471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690712,"kb_used_data":1040,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93681128,"statfs":{"total":96636764160,"available":95929475072,"internally_reserved":0,"allocated":1064960,"data_stored":851917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":2674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3072000,"data_stored":3072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1839104,"data_stored":1839104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11494", 'reported_seq': 11732, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.961050+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:49.961050+0000', 'last_peered': '2024-04-26T02:35:49.961050+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:49.961050+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1400", 'ondisk_log_start': "28'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 245760, 'num_objects': 30, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 30, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11494, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11818", 'reported_seq': 15365, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:51.588325+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:51.588325+0000', 'last_peered': '2024-04-26T02:35:51.588325+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:51.588325+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 111116, 'num_objects': 14, 'num_object_clones': 0, 'num_object_copies': 56, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 346, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 14, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11816, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1734, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:53.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11581", 'reported_seq': 11753, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:48.434245+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:48.434245+0000', 'last_peered': '2024-04-26T02:35:48.434245+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:48.434245+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 399646, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 50, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11580, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1934}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11578", 'reported_seq': 11814, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.850991+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:49.850991+0000', 'last_peered': '2024-04-26T02:35:49.850991+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:49.850991+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 152076, 'num_objects': 19, 'num_object_clones': 0, 'num_object_copies': 76, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 19, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11576, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1910}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11568", 'reported_seq': 11830, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.754826+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:49.754826+0000', 'last_peered': '2024-04-26T02:35:49.754826+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:49.754826+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 262144, 'num_objects': 32, 'num_object_clones': 0, 'num_object_copies': 128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 32, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11568, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11984", 'reported_seq': 12244, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:51.578568+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:51.578568+0000', 'last_peered': '2024-04-26T02:35:51.578568+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:51.578568+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37388, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11982, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11512", 'reported_seq': 11691, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:48.288402+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:48.288402+0000', 'last_peered': '2024-04-26T02:35:48.288402+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:48.288402+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 422412, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 52, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11510, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1943}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:53.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11691", 'reported_seq': 11931, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.784695+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:49.784695+0000', 'last_peered': '2024-04-26T02:35:49.784695+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:49.784695+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 224018, 'num_objects': 28, 'num_object_clones': 0, 'num_object_copies': 112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 28, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11688, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11572", 'reported_seq': 11807, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.960714+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:49.960714+0000', 'last_peered': '2024-04-26T02:35:49.960714+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:49.960714+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 203014, 'num_objects': 25, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 25, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11571, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1873}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11726", 'reported_seq': 11949, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:48.369395+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:48.369395+0000', 'last_peered': '2024-04-26T02:35:48.369395+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:48.369395+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 432390, 'num_objects': 53, 'num_object_clones': 0, 'num_object_copies': 212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 53, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11725, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12088", 'reported_seq': 12341, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:49.869615+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:49.869615+0000', 'last_peered': '2024-04-26T02:35:49.869615+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:49.869615+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "28'2000", 'ondisk_log_start': "28'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 274956, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 34, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12086, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1945}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:53.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11784", 'reported_seq': 12018, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:51.629540+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:51.629540+0000', 'last_peered': '2024-04-26T02:35:51.629540+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:51.629540+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 102924, 'num_objects': 13, 'num_object_clones': 0, 'num_object_copies': 52, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11782, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1974}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:53.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:53.766 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:53.766 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22042/1888 objects degraded (1167.479%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":750,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":472,"data_bytes":3808448,"bytes_used":2698170368,"bytes_avail":383848886272,"bytes_total":386547056640,"degraded_objects":22042,"degraded_total":1888,"degraded_ratio":11.67478813559322,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:35:53.786 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:53.786 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:54.131 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:54.132 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:54.150 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-04-26T02:35:52.771076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634932,"kb_used_data":11280,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374852428,"statfs":{"total":386547056640,"available":383848886272,"internally_reserved":0,"allocated":11550720,"data_stored":10698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,6,14,13,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17522688,"num_objects":-2139,"num_object_clones":0,"num_object_copies":-8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006868"},"pg_stats":[{"pgid":"1.f","version":"40'11494","reported_seq":11732,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.961050+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:49.961050+0000","last_peered":"2024-04-26T02:35:49.961050+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:49.961050+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760,"num_objects":30,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11494,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11581","reported_seq":11753,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.434245+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:48.434245+0000","last_peered":"2024-04-26T02:35:48.434245+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:48.434245+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399646,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11580,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11445","reported_seq":15476,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.107744+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:49.107744+0000","last_peered":"2024-04-26T02:35:49.107744+0000","last_clean":"2024-04-26T02:35:49.107744+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:49.107744+0000","last_undegraded":"2024-04-26T02:35:49.107744+0000","last_fullsized":"2024-04-26T02:35:49.107744+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283148,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11578","reported_seq":11814,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.850991+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:49.850991+0000","last_peered":"2024-04-26T02:35:49.850991+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:49.850991+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152076,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11568","reported_seq":11830,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.754826+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:49.754826+0000","last_peered":"2024-04-26T02:35:49.754826+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:49.754826+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11568,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11751","reported_seq":16009,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.271862+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:51.271862+0000","last_peered":"2024-04-26T02:35:51.271862+0000","last_clean":"2024-04-26T02:35:51.271862+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:51.271862+0000","last_undegraded":"2024-04-26T02:35:51.271862+0000","last_fullsized":"2024-04-26T02:35:51.271862+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:54.380420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53772,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11984","reported_seq":12244,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.578568+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:51.578568+0000","last_peered":"2024-04-26T02:35:51.578568+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:51.578568+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11705","reported_seq":15801,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.749587+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:49.749587+0000","last_peered":"2024-04-26T02:35:49.749587+0000","last_clean":"2024-04-26T02:35:49.749587+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:49.749587+0000","last_undegraded":"2024-04-26T02:35:49.749587+0000","last_fullsized":"2024-04-26T02:35:49.749587+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.260886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11512","reported_seq":11691,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.288402+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:48.288402+0000","last_peered":"2024-04-26T02:35:48.288402+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:48.288402+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422412,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11691","reported_seq":11931,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.784695+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:49.784695+0000","last_peered":"2024-04-26T02:35:49.784695+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:49.784695+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224018,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11572","reported_seq":11807,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.960714+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:49.960714+0000","last_peered":"2024-04-26T02:35:49.960714+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:49.960714+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203014,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1873}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11726","reported_seq":11949,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:48.369395+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:48.369395+0000","last_peered":"2024-04-26T02:35:48.369395+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:48.369395+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432390,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12088","reported_seq":12341,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:49.869615+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:49.869615+0000","last_peered":"2024-04-26T02:35:49.869615+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:49.869615+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"28'2000","ondisk_log_start":"28'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274956,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11815","reported_seq":16031,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:49.132122+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:49.132122+0000","last_peered":"2024-04-26T02:35:49.132122+0000","last_clean":"2024-04-26T02:35:49.132122+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:49.132122+0000","last_undegraded":"2024-04-26T02:35:49.132122+0000","last_fullsized":"2024-04-26T02:35:49.132122+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324108,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11784","reported_seq":12018,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.629540+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:51.629540+0000","last_peered":"2024-04-26T02:35:51.629540+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:51.629540+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102924,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1974}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3808448,"num_objects":472,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187085,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8134656,"data_stored":8134656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688276,"kb_used_data":2700,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683564,"statfs":{"total":96636764160,"available":95931969536,"internally_reserved":0,"allocated":2764800,"data_stored":2551757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689072,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93682768,"statfs":{"total":96636764160,"available":95931154432,"internally_reserved":0,"allocated":4104192,"data_stored":3891149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566872,"kb_used_data":3532,"kb_used_omap":4,"kb_used_meta":563323,"kb_avail":93804968,"statfs":{"total":96636764160,"available":96056287232,"internally_reserved":0,"allocated":3616768,"data_stored":3403725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":576843471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690712,"kb_used_data":1040,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93681128,"statfs":{"total":96636764160,"available":95929475072,"internally_reserved":0,"allocated":1064960,"data_stored":851917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706211535},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":2674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3072000,"data_stored":3072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1839104,"data_stored":1839104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:57.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:57.461 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:57.462 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:57.479 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-04-26T02:35:56.772372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625188,"kb_used_data":1536,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374862172,"statfs":{"total":386547056640,"available":383858864128,"internally_reserved":0,"allocated":1572864,"data_stored":711237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9764888,"num_objects":-1193,"num_object_clones":0,"num_object_copies":-4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006180"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11764,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923112+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:54.923112+0000","last_peered":"2024-04-26T02:35:54.923112+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:54.923112+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":11805,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.282929+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:52.282929+0000","last_peered":"2024-04-26T02:35:52.282929+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:52.282929+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15509,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.871588+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:51.871588+0000","last_peered":"2024-04-26T02:35:51.871588+0000","last_clean":"2024-04-26T02:35:51.871588+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:51.871588+0000","last_undegraded":"2024-04-26T02:35:51.871588+0000","last_fullsized":"2024-04-26T02:35:51.871588+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922684+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:54.922684+0000","last_peered":"2024-04-26T02:35:54.922684+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:54.922684+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922868+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:54.922868+0000","last_peered":"2024-04-26T02:35:54.922868+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:54.922868+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15834,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-26T02:35:54.922846+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:54.922846+0000","last_peered":"2024-04-26T02:35:54.922846+0000","last_clean":"2024-04-26T02:35:54.922846+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:54.922846+0000","last_undegraded":"2024-04-26T02:35:54.922846+0000","last_fullsized":"2024-04-26T02:35:54.922846+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11741,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.245265+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:52.245265+0000","last_peered":"2024-04-26T02:35:52.245265+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:52.245265+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923128+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:54.923128+0000","last_peered":"2024-04-26T02:35:54.923128+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:54.923128+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923242+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:54.923242+0000","last_peered":"2024-04-26T02:35:54.923242+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:54.923242+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12001,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.166411+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:52.166411+0000","last_peered":"2024-04-26T02:35:52.166411+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:52.166411+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12375,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922838+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:54.922838+0000","last_peered":"2024-04-26T02:35:54.922838+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:54.922838+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16069,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:52.038997+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:52.038997+0000","last_peered":"2024-04-26T02:35:52.038997+0000","last_clean":"2024-04-26T02:35:52.038997+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:52.038997+0000","last_undegraded":"2024-04-26T02:35:52.038997+0000","last_fullsized":"2024-04-26T02:35:52.038997+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685908,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685932,"statfs":{"total":96636764160,"available":95934394368,"internally_reserved":0,"allocated":339968,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689556,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682284,"statfs":{"total":96636764160,"available":95930658816,"internally_reserved":0,"allocated":339968,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555540,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93816300,"statfs":{"total":96636764160,"available":96067891200,"internally_reserved":0,"allocated":536576,"data_stored":323533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:57.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:58.621 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:58.622 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:58.644 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-04-26T02:35:56.772372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625188,"kb_used_data":1536,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374862172,"statfs":{"total":386547056640,"available":383858864128,"internally_reserved":0,"allocated":1572864,"data_stored":711237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9764888,"num_objects":-1193,"num_object_clones":0,"num_object_copies":-4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006180"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11764,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923112+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:54.923112+0000","last_peered":"2024-04-26T02:35:54.923112+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:54.923112+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":11805,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.282929+0000","last_change":"2024-04-26T02:24:59.565092+0000","last_active":"2024-04-26T02:35:52.282929+0000","last_peered":"2024-04-26T02:35:52.282929+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:52.282929+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15509,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.871588+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:51.871588+0000","last_peered":"2024-04-26T02:35:51.871588+0000","last_clean":"2024-04-26T02:35:51.871588+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:51.871588+0000","last_undegraded":"2024-04-26T02:35:51.871588+0000","last_fullsized":"2024-04-26T02:35:51.871588+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922684+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:54.922684+0000","last_peered":"2024-04-26T02:35:54.922684+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:54.922684+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922868+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:54.922868+0000","last_peered":"2024-04-26T02:35:54.922868+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:54.922868+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15834,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-26T02:35:54.922846+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:54.922846+0000","last_peered":"2024-04-26T02:35:54.922846+0000","last_clean":"2024-04-26T02:35:54.922846+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:54.922846+0000","last_undegraded":"2024-04-26T02:35:54.922846+0000","last_fullsized":"2024-04-26T02:35:54.922846+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11741,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.245265+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:52.245265+0000","last_peered":"2024-04-26T02:35:52.245265+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:52.245265+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923128+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:54.923128+0000","last_peered":"2024-04-26T02:35:54.923128+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:54.923128+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923242+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:54.923242+0000","last_peered":"2024-04-26T02:35:54.923242+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:54.923242+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12001,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:52.166411+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:52.166411+0000","last_peered":"2024-04-26T02:35:52.166411+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:52.166411+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12375,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922838+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:54.922838+0000","last_peered":"2024-04-26T02:35:54.922838+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:54.922838+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16069,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:52.038997+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:52.038997+0000","last_peered":"2024-04-26T02:35:52.038997+0000","last_clean":"2024-04-26T02:35:52.038997+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:52.038997+0000","last_undegraded":"2024-04-26T02:35:52.038997+0000","last_fullsized":"2024-04-26T02:35:52.038997+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685908,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685932,"statfs":{"total":96636764160,"available":95934394368,"internally_reserved":0,"allocated":339968,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689556,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682284,"statfs":{"total":96636764160,"available":95930658816,"internally_reserved":0,"allocated":339968,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555540,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93816300,"statfs":{"total":96636764160,"available":96067891200,"internally_reserved":0,"allocated":536576,"data_stored":323533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11764, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.923112+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:54.923112+0000', 'last_peered': '2024-04-26T02:35:54.923112+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:54.923112+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "40'11818", 'reported_seq': 15365, 'reported_epoch': 40, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:51.588325+0000', 'last_change': '2024-04-26T02:33:23.026229+0000', 'last_active': '2024-04-26T02:35:51.588325+0000', 'last_peered': '2024-04-26T02:35:51.588325+0000', 'last_clean': '2024-04-26T02:23:50.934410+0000', 'last_became_active': '2024-04-26T02:24:59.558275+0000', 'last_became_peered': '2024-04-26T02:24:59.558275+0000', 'last_unstale': '2024-04-26T02:35:51.588325+0000', 'last_undegraded': '2024-04-26T02:24:59.256651+0000', 'last_fullsized': '2024-04-26T02:24:59.235746+0000', 'mapping_epoch': 31, 'log_start': "28'1800", 'ondisk_log_start': "28'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 111116, 'num_objects': 14, 'num_object_clones': 0, 'num_object_copies': 56, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 346, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 14, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11816, 'num_write_kb': 47324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1734, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11630", 'reported_seq': 11805, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:52.282929+0000', 'last_change': '2024-04-26T02:24:59.565092+0000', 'last_active': '2024-04-26T02:35:52.282929+0000', 'last_peered': '2024-04-26T02:35:52.282929+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:52.282929+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11629, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1983}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:35:58.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11833, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.922684+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:54.922684+0000', 'last_peered': '2024-04-26T02:35:54.922684+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:54.922684+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11864, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.922868+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:54.922868+0000', 'last_peered': '2024-04-26T02:35:54.922868+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:54.922868+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 12251, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921004+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:55.921004+0000', 'last_peered': '2024-04-26T02:35:55.921004+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:55.921004+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11741, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:52.245265+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:52.245265+0000', 'last_peered': '2024-04-26T02:35:52.245265+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:52.245265+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11958, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.923128+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:54.923128+0000', 'last_peered': '2024-04-26T02:35:54.923128+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:54.923128+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:35:58.646 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11833, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.923242+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:54.923242+0000', 'last_peered': '2024-04-26T02:35:54.923242+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:54.923242+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12001, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:52.166411+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:52.166411+0000', 'last_peered': '2024-04-26T02:35:52.166411+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:52.166411+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12375, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:54.922838+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:54.922838+0000', 'last_peered': '2024-04-26T02:35:54.922838+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:54.922838+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:35:58.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12034, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921088+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:55.921088+0000', 'last_peered': '2024-04-26T02:35:55.921088+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:55.921088+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:35:58.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:35:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:59.011 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21448/160 objects degraded (13405.000%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":755,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":40,"data_bytes":277672,"bytes_used":2688212992,"bytes_avail":383858843648,"bytes_total":386547056640,"degraded_objects":21448,"degraded_total":160,"degraded_ratio":134.05000000000001,"recovering_objects_per_sec":80,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":85},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:35:59.030 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:35:59.031 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:35:59.336 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:35:59.336 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:35:59.355 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-04-26T02:35:58.772745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11321,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2625208,"kb_used_data":1552,"kb_used_omap":14,"kb_used_meta":2623601,"kb_avail":374862152,"statfs":{"total":386547056640,"available":383858843648,"internally_reserved":0,"allocated":1589248,"data_stored":718166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2686567987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5578776,"num_objects":-682,"num_object_clones":0,"num_object_copies":-2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11764,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923112+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:54.923112+0000","last_peered":"2024-04-26T02:35:54.923112+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:54.923112+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11818","reported_seq":15365,"reported_epoch":40,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:51.588325+0000","last_change":"2024-04-26T02:33:23.026229+0000","last_active":"2024-04-26T02:35:51.588325+0000","last_peered":"2024-04-26T02:35:51.588325+0000","last_clean":"2024-04-26T02:23:50.934410+0000","last_became_active":"2024-04-26T02:24:59.558275+0000","last_became_peered":"2024-04-26T02:24:59.558275+0000","last_unstale":"2024-04-26T02:35:51.588325+0000","last_undegraded":"2024-04-26T02:24:59.256651+0000","last_fullsized":"2024-04-26T02:24:59.235746+0000","mapping_epoch":31,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111116,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":12998,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:58.468291+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:35:58.468291+0000","last_peered":"2024-04-26T02:35:58.468291+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:58.468291+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15509,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:51.871588+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:51.871588+0000","last_peered":"2024-04-26T02:35:51.871588+0000","last_clean":"2024-04-26T02:35:51.871588+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:51.871588+0000","last_undegraded":"2024-04-26T02:35:51.871588+0000","last_fullsized":"2024-04-26T02:35:51.871588+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.397549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922684+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:54.922684+0000","last_peered":"2024-04-26T02:35:54.922684+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:54.922684+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11864,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922868+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:54.922868+0000","last_peered":"2024-04-26T02:35:54.922868+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:54.922868+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15834,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-26T02:35:54.922846+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:54.922846+0000","last_peered":"2024-04-26T02:35:54.922846+0000","last_clean":"2024-04-26T02:35:54.922846+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:54.922846+0000","last_undegraded":"2024-04-26T02:35:54.922846+0000","last_fullsized":"2024-04-26T02:35:54.922846+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11958,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923128+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:54.923128+0000","last_peered":"2024-04-26T02:35:54.923128+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:54.923128+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11833,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.923242+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:54.923242+0000","last_peered":"2024-04-26T02:35:54.923242+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:54.923242+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12375,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:54.922838+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:54.922838+0000","last_peered":"2024-04-26T02:35:54.922838+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:54.922838+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16069,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-26T02:35:52.038997+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:52.038997+0000","last_peered":"2024-04-26T02:35:52.038997+0000","last_clean":"2024-04-26T02:35:52.038997+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:52.038997+0000","last_undegraded":"2024-04-26T02:35:52.038997+0000","last_fullsized":"2024-04-26T02:35:52.038997+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:52.356175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":277672,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21448,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187517,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11321,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685908,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685932,"statfs":{"total":96636764160,"available":95934394368,"internally_reserved":0,"allocated":339968,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019011,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555540,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93816300,"statfs":{"total":96636764160,"available":96067891200,"internally_reserved":0,"allocated":536576,"data_stored":323533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:02.357 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:02.704 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:02.705 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:02.722 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-04-26T02:36:00.773194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647448,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839912,"statfs":{"total":386547056640,"available":383836069888,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3629080,"num_objects":-444,"num_object_clones":0,"num_object_copies":-1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002118"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":12998,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:58.468291+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:35:58.468291+0000","last_peered":"2024-04-26T02:35:58.468291+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:58.468291+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577760,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794080,"statfs":{"total":96636764160,"available":96045137920,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:02.723 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:03.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:03.045 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:03.063 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-26T02:36:02.773937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647448,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839912,"statfs":{"total":386547056640,"available":383836069888,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1818624,"num_objects":-222,"num_object_clones":0,"num_object_copies":-888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002304"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":12998,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:58.468291+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:35:58.468291+0000","last_peered":"2024-04-26T02:35:58.468291+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:58.468291+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577760,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794080,"statfs":{"total":96636764160,"available":96045137920,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11766, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926197+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:55.926197+0000', 'last_peered': '2024-04-26T02:35:55.926197+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:55.926197+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11630", 'reported_seq': 12998, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:58.468291+0000', 'last_change': '2024-04-26T02:35:53.902690+0000', 'last_active': '2024-04-26T02:35:58.468291+0000', 'last_peered': '2024-04-26T02:35:58.468291+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:35:58.468291+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1391, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11629, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 635, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1390}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.924662+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:55.924662+0000', 'last_peered': '2024-04-26T02:35:55.924662+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:55.924662+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11866, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.927821+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:55.927821+0000', 'last_peered': '2024-04-26T02:35:55.927821+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:55.927821+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:03.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 12251, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921004+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:55.921004+0000', 'last_peered': '2024-04-26T02:35:55.921004+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:55.921004+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11745, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.922143+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:55.922143+0000', 'last_peered': '2024-04-26T02:35:55.922143+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:55.922143+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11960, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926161+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:55.926161+0000', 'last_peered': '2024-04-26T02:35:55.926161+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:55.926161+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926283+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:55.926283+0000', 'last_peered': '2024-04-26T02:35:55.926283+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:55.926283+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12005, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.915575+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:55.915575+0000', 'last_peered': '2024-04-26T02:35:55.915575+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:55.915575+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:03.065 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12377, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.925275+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:55.925275+0000', 'last_peered': '2024-04-26T02:35:55.925275+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:55.925275+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:03.066 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:03.066 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12034, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921088+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:55.921088+0000', 'last_peered': '2024-04-26T02:35:55.921088+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:55.921088+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:03.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:03.426 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:03.426 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21102/112 objects degraded (18841.071%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":759,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2710986752,"bytes_avail":383836069888,"bytes_total":386547056640,"degraded_objects":21102,"degraded_total":112,"degraded_ratio":188.41071428571428,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":27},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:03.444 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:03.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:03.746 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:03.747 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:03.764 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-26T02:36:02.773937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647448,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839912,"statfs":{"total":386547056640,"available":383836069888,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1818624,"num_objects":-222,"num_object_clones":0,"num_object_copies":-888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002304"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":12998,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:58.468291+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:35:58.468291+0000","last_peered":"2024-04-26T02:35:58.468291+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:35:58.468291+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11667,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577760,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794080,"statfs":{"total":96636764160,"available":96045137920,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:06.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:07.094 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:07.095 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:07.113 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-04-26T02:36:06.775336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651608,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835752,"statfs":{"total":386547056640,"available":383831810048,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-253952,"num_objects":-31,"num_object_clones":0,"num_object_copies":-124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.860533"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":13956,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:03.477399+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:03.477399+0000","last_peered":"2024-04-26T02:36:03.477399+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:03.477399+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581920,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789920,"statfs":{"total":96636764160,"available":96040878080,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:07.115 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:07.421 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:07.421 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:07.439 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-04-26T02:36:06.775336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651608,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835752,"statfs":{"total":386547056640,"available":383831810048,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-253952,"num_objects":-31,"num_object_clones":0,"num_object_copies":-124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.860533"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":13956,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:03.477399+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:03.477399+0000","last_peered":"2024-04-26T02:36:03.477399+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:03.477399+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581920,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789920,"statfs":{"total":96636764160,"available":96040878080,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:07.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11766, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926197+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:55.926197+0000', 'last_peered': '2024-04-26T02:35:55.926197+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:55.926197+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11630", 'reported_seq': 13956, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:03.477399+0000', 'last_change': '2024-04-26T02:35:53.902690+0000', 'last_active': '2024-04-26T02:36:03.477399+0000', 'last_peered': '2024-04-26T02:36:03.477399+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:36:03.477399+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11629, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1114, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -911}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.924662+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:55.924662+0000', 'last_peered': '2024-04-26T02:35:55.924662+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:55.924662+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11866, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.927821+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:55.927821+0000', 'last_peered': '2024-04-26T02:35:55.927821+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:55.927821+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:07.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 12251, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921004+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:55.921004+0000', 'last_peered': '2024-04-26T02:35:55.921004+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:55.921004+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11745, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.922143+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:55.922143+0000', 'last_peered': '2024-04-26T02:35:55.922143+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:55.922143+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11960, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926161+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:55.926161+0000', 'last_peered': '2024-04-26T02:35:55.926161+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:55.926161+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926283+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:55.926283+0000', 'last_peered': '2024-04-26T02:35:55.926283+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:55.926283+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12005, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.915575+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:55.915575+0000', 'last_peered': '2024-04-26T02:35:55.915575+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:55.915575+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12377, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.925275+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:55.925275+0000', 'last_peered': '2024-04-26T02:35:55.925275+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:55.925275+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:07.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:07.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12034, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921088+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:55.921088+0000', 'last_peered': '2024-04-26T02:35:55.921088+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:55.921088+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:07.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:07.826 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20623/112 objects degraded (18413.393%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":764,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715246592,"bytes_avail":383831810048,"bytes_total":386547056640,"degraded_objects":20623,"degraded_total":112,"degraded_ratio":184.13392857142858,"recovering_objects_per_sec":130,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1419,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:07.844 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:07.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:08.132 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:08.132 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:08.149 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-04-26T02:36:06.775336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651608,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835752,"statfs":{"total":386547056640,"available":383831810048,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-253952,"num_objects":-31,"num_object_clones":0,"num_object_copies":-124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.860533"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":13956,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:03.477399+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:03.477399+0000","last_peered":"2024-04-26T02:36:03.477399+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:03.477399+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12146,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581920,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789920,"statfs":{"total":96636764160,"available":96040878080,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:11.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:11.498 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:11.498 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:11.518 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-26T02:36:11.002269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12793,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643288,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374844072,"statfs":{"total":386547056640,"available":383840329728,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.229074"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15250,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:08.474680+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:08.474680+0000","last_peered":"2024-04-26T02:36:08.474680+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:08.474680+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12793,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573600,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798240,"statfs":{"total":96636764160,"available":96049397760,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:11.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:11.879 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:11.880 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:11.900 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-26T02:36:11.002269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12793,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643288,"kb_used_data":1580,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374844072,"statfs":{"total":386547056640,"available":383840329728,"internally_reserved":0,"allocated":1617920,"data_stored":727928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.229074"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15250,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:08.474680+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:08.474680+0000","last_peered":"2024-04-26T02:36:08.474680+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:08.474680+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16018,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920870+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:35:55.920870+0000","last_peered":"2024-04-26T02:35:55.920870+0000","last_clean":"2024-04-26T02:35:55.920870+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:35:55.920870+0000","last_undegraded":"2024-04-26T02:35:55.920870+0000","last_fullsized":"2024-04-26T02:35:55.920870+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:59.408775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921004+0000","last_change":"2024-04-26T02:24:59.638327+0000","last_active":"2024-04-26T02:35:55.921004+0000","last_peered":"2024-04-26T02:35:55.921004+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:35:55.921004+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.921088+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:35:55.921088+0000","last_peered":"2024-04-26T02:35:55.921088+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:35:55.921088+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12793,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573600,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798240,"statfs":{"total":96636764160,"available":96049397760,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694184,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677656,"statfs":{"total":96636764160,"available":95925919744,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:11.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:11.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11766, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926197+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:55.926197+0000', 'last_peered': '2024-04-26T02:35:55.926197+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:55.926197+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-04-26T02:36:11.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "40'11630", 'reported_seq': 15250, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:08.474680+0000', 'last_change': '2024-04-26T02:35:53.902690+0000', 'last_active': '2024-04-26T02:36:08.474680+0000', 'last_peered': '2024-04-26T02:36:08.474680+0000', 'last_clean': '2024-04-26T02:23:51.346443+0000', 'last_became_active': '2024-04-26T02:24:59.555816+0000', 'last_became_peered': '2024-04-26T02:24:59.555816+0000', 'last_unstale': '2024-04-26T02:36:08.474680+0000', 'last_undegraded': '2024-04-26T02:24:59.266666+0000', 'last_fullsized': '2024-04-26T02:24:59.230659+0000', 'mapping_epoch': 31, 'log_start': "28'1600", 'ondisk_log_start': "28'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11629, 'num_write_kb': 46511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1761, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -264}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:11.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.924662+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:55.924662+0000', 'last_peered': '2024-04-26T02:35:55.924662+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:55.924662+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11866, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.927821+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:55.927821+0000', 'last_peered': '2024-04-26T02:35:55.927821+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:55.927821+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 12251, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921004+0000', 'last_change': '2024-04-26T02:24:59.638327+0000', 'last_active': '2024-04-26T02:35:55.921004+0000', 'last_peered': '2024-04-26T02:35:55.921004+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:35:55.921004+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11745, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.922143+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:35:55.922143+0000', 'last_peered': '2024-04-26T02:35:55.922143+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:35:55.922143+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11960, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926161+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:55.926161+0000', 'last_peered': '2024-04-26T02:35:55.926161+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:55.926161+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926283+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:55.926283+0000', 'last_peered': '2024-04-26T02:35:55.926283+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:55.926283+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12005, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.915575+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:35:55.915575+0000', 'last_peered': '2024-04-26T02:35:55.915575+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:35:55.915575+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12377, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.925275+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:55.925275+0000', 'last_peered': '2024-04-26T02:35:55.925275+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:55.925275+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:11.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12034, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.921088+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:35:55.921088+0000', 'last_peered': '2024-04-26T02:35:55.921088+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:35:55.921088+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:11.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:12.354 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:12.354 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19976/112 objects degraded (17835.714%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":768,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2706726912,"bytes_avail":383840329728,"bytes_total":386547056640,"degraded_objects":19976,"degraded_total":112,"degraded_ratio":178.35714285714286,"recovering_objects_per_sec":122,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1472,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:12.373 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:12.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:12.671 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:12.671 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:12.688 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-04-26T02:36:12.001931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12915,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643296,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374844064,"statfs":{"total":386547056640,"available":383840321536,"internally_reserved":0,"allocated":1626112,"data_stored":731366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.227993"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15407,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:56.049897+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:35:56.049897+0000","last_peered":"2024-04-26T02:35:56.049897+0000","last_clean":"2024-04-26T02:35:56.049897+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:35:56.049897+0000","last_undegraded":"2024-04-26T02:35:56.049897+0000","last_fullsized":"2024-04-26T02:35:56.049897+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:01.367752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15250,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:08.474680+0000","last_change":"2024-04-26T02:35:53.902690+0000","last_active":"2024-04-26T02:36:08.474680+0000","last_peered":"2024-04-26T02:36:08.474680+0000","last_clean":"2024-04-26T02:23:51.346443+0000","last_became_active":"2024-04-26T02:24:59.555816+0000","last_became_peered":"2024-04-26T02:24:59.555816+0000","last_unstale":"2024-04-26T02:36:08.474680+0000","last_undegraded":"2024-04-26T02:24:59.266666+0000","last_fullsized":"2024-04-26T02:24:59.230659+0000","mapping_epoch":31,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15513,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.920706+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:35:55.920706+0000","last_peered":"2024-04-26T02:35:55.920706+0000","last_clean":"2024-04-26T02:35:55.920706+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:35:55.920706+0000","last_undegraded":"2024-04-26T02:35:55.920706+0000","last_fullsized":"2024-04-26T02:35:55.920706+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16020,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-26T02:36:11.023529+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:11.023529+0000","last_peered":"2024-04-26T02:36:11.023529+0000","last_clean":"2024-04-26T02:36:11.023529+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:11.023529+0000","last_undegraded":"2024-04-26T02:36:11.023529+0000","last_fullsized":"2024-04-26T02:36:11.023529+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12498,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.635695+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:11.635695+0000","last_peered":"2024-04-26T02:36:11.635695+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:11.635695+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11745,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.922143+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:35:55.922143+0000","last_peered":"2024-04-26T02:35:55.922143+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:35:55.922143+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12005,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.915575+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:35:55.915575+0000","last_peered":"2024-04-26T02:35:55.915575+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:35:55.915575+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16073,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.922186+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:35:55.922186+0000","last_peered":"2024-04-26T02:35:55.922186+0000","last_clean":"2024-04-26T02:35:55.922186+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:35:55.922186+0000","last_undegraded":"2024-04-26T02:35:55.922186+0000","last_fullsized":"2024-04-26T02:35:55.922186+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.427363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.028225+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:11.028225+0000","last_peered":"2024-04-26T02:36:11.028225+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:11.028225+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12915,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689576,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682264,"statfs":{"total":96636764160,"available":95930638336,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573600,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798240,"statfs":{"total":96636764160,"available":96049397760,"internally_reserved":0,"allocated":548864,"data_stored":326366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694192,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677648,"statfs":{"total":96636764160,"available":95925911552,"internally_reserved":0,"allocated":364544,"data_stored":137292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:15.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:16.005 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:16.005 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:16.025 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-04-26T02:36:14.778441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13180,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647484,"kb_used_data":1612,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839876,"statfs":{"total":386547056640,"available":383836033024,"internally_reserved":0,"allocated":1650688,"data_stored":736884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1034,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003104"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16020,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-26T02:36:11.023529+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:11.023529+0000","last_peered":"2024-04-26T02:36:11.023529+0000","last_clean":"2024-04-26T02:36:11.023529+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:11.023529+0000","last_undegraded":"2024-04-26T02:36:11.023529+0000","last_fullsized":"2024-04-26T02:36:11.023529+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12498,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.635695+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:11.635695+0000","last_peered":"2024-04-26T02:36:11.635695+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:11.635695+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11749,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.004191+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.004191+0000","last_peered":"2024-04-26T02:36:12.004191+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.004191+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.028225+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:11.028225+0000","last_peered":"2024-04-26T02:36:11.028225+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:11.028225+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13180,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577768,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794072,"statfs":{"total":96636764160,"available":96045129728,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694192,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677648,"statfs":{"total":96636764160,"available":95925911552,"internally_reserved":0,"allocated":364544,"data_stored":137292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:16.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:16.326 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:16.327 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:16.346 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-04-26T02:36:14.778441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13180,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647484,"kb_used_data":1612,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839876,"statfs":{"total":386547056640,"available":383836033024,"internally_reserved":0,"allocated":1650688,"data_stored":736884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1034,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003104"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11766,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926197+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:35:55.926197+0000","last_peered":"2024-04-26T02:35:55.926197+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:35:55.926197+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.924662+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:35:55.924662+0000","last_peered":"2024-04-26T02:35:55.924662+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:35:55.924662+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11866,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.927821+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:35:55.927821+0000","last_peered":"2024-04-26T02:35:55.927821+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:35:55.927821+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16020,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-26T02:36:11.023529+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:11.023529+0000","last_peered":"2024-04-26T02:36:11.023529+0000","last_clean":"2024-04-26T02:36:11.023529+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:11.023529+0000","last_undegraded":"2024-04-26T02:36:11.023529+0000","last_fullsized":"2024-04-26T02:36:11.023529+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12498,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.635695+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:11.635695+0000","last_peered":"2024-04-26T02:36:11.635695+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:11.635695+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15836,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-26T02:35:55.927700+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:35:55.927700+0000","last_peered":"2024-04-26T02:35:55.927700+0000","last_clean":"2024-04-26T02:35:55.927700+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:35:55.927700+0000","last_undegraded":"2024-04-26T02:35:55.927700+0000","last_fullsized":"2024-04-26T02:35:55.927700+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:57.328019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11749,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.004191+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.004191+0000","last_peered":"2024-04-26T02:36:12.004191+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.004191+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926161+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:35:55.926161+0000","last_peered":"2024-04-26T02:35:55.926161+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:35:55.926161+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11835,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.926283+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:35:55.926283+0000","last_peered":"2024-04-26T02:35:55.926283+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:35:55.926283+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:35:55.925275+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:35:55.925275+0000","last_peered":"2024-04-26T02:35:55.925275+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:55.925275+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:11.028225+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:11.028225+0000","last_peered":"2024-04-26T02:36:11.028225+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:11.028225+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13180,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685928,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685912,"statfs":{"total":96636764160,"available":95934373888,"internally_reserved":0,"allocated":356352,"data_stored":133854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577768,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794072,"statfs":{"total":96636764160,"available":96045129728,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694192,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677648,"statfs":{"total":96636764160,"available":95925911552,"internally_reserved":0,"allocated":364544,"data_stored":137292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:16.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:16.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11766, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926197+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:35:55.926197+0000', 'last_peered': '2024-04-26T02:35:55.926197+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:35:55.926197+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:16.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.924662+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:35:55.924662+0000', 'last_peered': '2024-04-26T02:35:55.924662+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:35:55.924662+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11866, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.927821+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:35:55.927821+0000', 'last_peered': '2024-04-26T02:35:55.927821+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:35:55.927821+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 12498, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:11.635695+0000', 'last_change': '2024-04-26T02:36:10.613092+0000', 'last_active': '2024-04-26T02:36:11.635695+0000', 'last_peered': '2024-04-26T02:36:11.635695+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:36:11.635695+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1932, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 213, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11749, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.004191+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:12.004191+0000', 'last_peered': '2024-04-26T02:36:12.004191+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:12.004191+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11960, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926161+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:35:55.926161+0000', 'last_peered': '2024-04-26T02:35:55.926161+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:35:55.926161+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:16.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11835, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.926283+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:35:55.926283+0000', 'last_peered': '2024-04-26T02:35:55.926283+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:35:55.926283+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12009, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.003303+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:12.003303+0000', 'last_peered': '2024-04-26T02:36:12.003303+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:12.003303+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12377, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:35:55.925275+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:35:55.925275+0000', 'last_peered': '2024-04-26T02:35:55.925275+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:55.925275+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:16.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12036, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:11.028225+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:11.028225+0000', 'last_peered': '2024-04-26T02:36:11.028225+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:11.028225+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:16.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:16.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:16.727 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19589/112 objects degraded (17490.179%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":773,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711023616,"bytes_avail":383836033024,"bytes_total":386547056640,"degraded_objects":19589,"degraded_total":112,"degraded_ratio":174.90178571428572,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1034,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:16.744 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:16.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:17.036 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:17.054 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-04-26T02:36:16.778997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647516,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839844,"statfs":{"total":386547056640,"available":383836000256,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002897"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":13412,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:16.638636+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:16.638636+0000","last_peered":"2024-04-26T02:36:16.638636+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:16.638636+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11749,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.004191+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.004191+0000","last_peered":"2024-04-26T02:36:12.004191+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.004191+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577768,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794072,"statfs":{"total":96636764160,"available":96045129728,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:20.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:20.376 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:20.377 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:20.395 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-04-26T02:36:18.779583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647516,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839844,"statfs":{"total":386547056640,"available":383836000256,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002721"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":13412,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:16.638636+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:16.638636+0000","last_peered":"2024-04-26T02:36:16.638636+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:16.638636+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577768,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794072,"statfs":{"total":96636764160,"available":96045129728,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:20.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:20.698 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:20.717 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-04-26T02:36:18.779583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647516,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839844,"statfs":{"total":386547056640,"available":383836000256,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002721"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":13412,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:16.638636+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:16.638636+0000","last_peered":"2024-04-26T02:36:16.638636+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:16.638636+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577768,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794072,"statfs":{"total":96636764160,"available":96045129728,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11770, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009476+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:36:12.009476+0000', 'last_peered': '2024-04-26T02:36:12.009476+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:12.009476+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006739+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:12.006739+0000', 'last_peered': '2024-04-26T02:36:12.006739+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:12.006739+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:20.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11870, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.014057+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:12.014057+0000', 'last_peered': '2024-04-26T02:36:12.014057+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:12.014057+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 13412, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:16.638636+0000', 'last_change': '2024-04-26T02:36:10.613092+0000', 'last_active': '2024-04-26T02:36:16.638636+0000', 'last_peered': '2024-04-26T02:36:16.638636+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:36:16.638636+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 669, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.005339+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:12.005339+0000', 'last_peered': '2024-04-26T02:36:12.005339+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:12.005339+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11964, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009493+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:12.009493+0000', 'last_peered': '2024-04-26T02:36:12.009493+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:12.009493+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.011774+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:12.011774+0000', 'last_peered': '2024-04-26T02:36:12.011774+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:12.011774+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:20.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12009, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.003303+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:12.003303+0000', 'last_peered': '2024-04-26T02:36:12.003303+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:12.003303+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:20.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:20.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12381, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006820+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:12.006820+0000', 'last_peered': '2024-04-26T02:36:12.006820+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:12.006820+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:20.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:20.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12038, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009592+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:12.009592+0000', 'last_peered': '2024-04-26T02:36:12.009592+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:12.009592+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:20.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:21.088 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19133/112 objects degraded (17083.036%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":777,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715316224,"bytes_avail":383831740416,"bytes_total":386547056640,"degraded_objects":19133,"degraded_total":112,"degraded_ratio":170.83035714285714,"recovering_objects_per_sec":86,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:21.107 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:21.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:21.405 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:21.405 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:21.424 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-04-26T02:36:20.780269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.777999"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":13412,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:16.638636+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:16.638636+0000","last_peered":"2024-04-26T02:36:16.638636+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:16.638636+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13636,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:24.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:24.772 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:24.789 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-04-26T02:36:22.780793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.778862"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":14328,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:21.636511+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:21.636511+0000","last_peered":"2024-04-26T02:36:21.636511+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:21.636511+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019016,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:24.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:25.101 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:25.102 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:25.120 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-04-26T02:36:24.781483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003739"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":14328,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:21.636511+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:21.636511+0000","last_peered":"2024-04-26T02:36:21.636511+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:21.636511+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:25.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:25.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11770, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009476+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:36:12.009476+0000', 'last_peered': '2024-04-26T02:36:12.009476+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:12.009476+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:25.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006739+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:12.006739+0000', 'last_peered': '2024-04-26T02:36:12.006739+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:12.006739+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11870, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.014057+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:12.014057+0000', 'last_peered': '2024-04-26T02:36:12.014057+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:12.014057+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 14328, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:21.636511+0000', 'last_change': '2024-04-26T02:36:10.613092+0000', 'last_active': '2024-04-26T02:36:21.636511+0000', 'last_peered': '2024-04-26T02:36:21.636511+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:36:21.636511+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1018, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1016}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:25.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.005339+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:12.005339+0000', 'last_peered': '2024-04-26T02:36:12.005339+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:12.005339+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:25.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:25.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11964, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009493+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:12.009493+0000', 'last_peered': '2024-04-26T02:36:12.009493+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:12.009493+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:25.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.011774+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:12.011774+0000', 'last_peered': '2024-04-26T02:36:12.011774+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:12.011774+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:25.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12009, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.003303+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:12.003303+0000', 'last_peered': '2024-04-26T02:36:12.003303+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:12.003303+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:25.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:25.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12381, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006820+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:12.006820+0000', 'last_peered': '2024-04-26T02:36:12.006820+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:12.006820+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:25.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:25.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12038, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009592+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:12.009592+0000', 'last_peered': '2024-04-26T02:36:12.009592+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:12.009592+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:25.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:25.548 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:25.549 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18675/112 objects degraded (16674.107%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":782,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715316224,"bytes_avail":383831740416,"bytes_total":386547056640,"degraded_objects":18675,"degraded_total":112,"degraded_ratio":166.74107142857142,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1179,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:25.568 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:25.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:25.874 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:25.891 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-04-26T02:36:24.781483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003739"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006739+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.006739+0000","last_peered":"2024-04-26T02:36:12.006739+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.006739+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":14328,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:21.636511+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:21.636511+0000","last_peered":"2024-04-26T02:36:21.636511+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:21.636511+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14094,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:28.892 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:29.213 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:29.213 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:29.231 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-04-26T02:36:28.782944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003947"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:29.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:29.532 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:29.532 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:29.551 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-04-26T02:36:28.782944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003947"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:29.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:29.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11770, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009476+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:36:12.009476+0000', 'last_peered': '2024-04-26T02:36:12.009476+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:12.009476+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11840, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.007941+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:12.007941+0000', 'last_peered': '2024-04-26T02:36:12.007941+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:12.007941+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11870, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.014057+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:12.014057+0000', 'last_peered': '2024-04-26T02:36:12.014057+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:12.014057+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 15364, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:26.637953+0000', 'last_change': '2024-04-26T02:36:10.613092+0000', 'last_active': '2024-04-26T02:36:26.637953+0000', 'last_peered': '2024-04-26T02:36:26.637953+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:36:26.637953+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 500, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1645, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.005339+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:12.005339+0000', 'last_peered': '2024-04-26T02:36:12.005339+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:12.005339+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:29.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11964, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009493+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:12.009493+0000', 'last_peered': '2024-04-26T02:36:12.009493+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:12.009493+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.011774+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:12.011774+0000', 'last_peered': '2024-04-26T02:36:12.011774+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:12.011774+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12009, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.003303+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:12.003303+0000', 'last_peered': '2024-04-26T02:36:12.003303+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:12.003303+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12381, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006820+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:12.006820+0000', 'last_peered': '2024-04-26T02:36:12.006820+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:12.006820+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:29.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12038, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009592+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:12.009592+0000', 'last_peered': '2024-04-26T02:36:12.009592+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:12.009592+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:29.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:29.924 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:29.924 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18157/112 objects degraded (16211.607%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":786,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715316224,"bytes_avail":383831740416,"bytes_total":386547056640,"degraded_objects":18157,"degraded_total":112,"degraded_ratio":162.11607142857142,"recovering_objects_per_sec":81,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:29.941 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:29.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:30.228 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:30.228 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:30.244 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-04-26T02:36:28.782944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651676,"kb_used_data":1636,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835684,"statfs":{"total":386547056640,"available":383831740416,"internally_reserved":0,"allocated":1675264,"data_stored":747156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003947"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16022,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.007686+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:12.007686+0000","last_peered":"2024-04-26T02:36:12.007686+0000","last_clean":"2024-04-26T02:36:12.007686+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:12.007686+0000","last_undegraded":"2024-04-26T02:36:12.007686+0000","last_fullsized":"2024-04-26T02:36:12.007686+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:14.581488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12038,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009592+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:12.009592+0000","last_peered":"2024-04-26T02:36:12.009592+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:12.009592+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581928,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789912,"statfs":{"total":96636764160,"available":96040869888,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694204,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677636,"statfs":{"total":96636764160,"available":95925899264,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:33.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:33.567 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:33.567 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:33.585 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-04-26T02:36:32.784534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639200,"kb_used_data":1644,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848160,"statfs":{"total":386547056640,"available":383844515840,"internally_reserved":0,"allocated":1683456,"data_stored":750557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003050"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16024,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-26T02:36:31.095323+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:31.095323+0000","last_peered":"2024-04-26T02:36:31.095323+0000","last_clean":"2024-04-26T02:36:31.095323+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:31.095323+0000","last_undegraded":"2024-04-26T02:36:31.095323+0000","last_fullsized":"2024-04-26T02:36:31.095323+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12040,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:31.102424+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:31.102424+0000","last_peered":"2024-04-26T02:36:31.102424+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:31.102424+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569448,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802392,"statfs":{"total":96636764160,"available":96053649408,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694208,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677632,"statfs":{"total":96636764160,"available":95925895168,"internally_reserved":0,"allocated":380928,"data_stored":144110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:33.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:33.928 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:33.946 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-04-26T02:36:32.784534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639200,"kb_used_data":1644,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848160,"statfs":{"total":386547056640,"available":383844515840,"internally_reserved":0,"allocated":1683456,"data_stored":750557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003050"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16024,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-26T02:36:31.095323+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:31.095323+0000","last_peered":"2024-04-26T02:36:31.095323+0000","last_clean":"2024-04-26T02:36:31.095323+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:31.095323+0000","last_undegraded":"2024-04-26T02:36:31.095323+0000","last_fullsized":"2024-04-26T02:36:31.095323+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12040,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:31.102424+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:31.102424+0000","last_peered":"2024-04-26T02:36:31.102424+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:31.102424+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569448,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802392,"statfs":{"total":96636764160,"available":96053649408,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694208,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677632,"statfs":{"total":96636764160,"available":95925895168,"internally_reserved":0,"allocated":380928,"data_stored":144110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:33.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 11770, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009476+0000', 'last_change': '2024-04-26T02:24:59.581976+0000', 'last_active': '2024-04-26T02:36:12.009476+0000', 'last_peered': '2024-04-26T02:36:12.009476+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:12.009476+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 90, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11840, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.007941+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:12.007941+0000', 'last_peered': '2024-04-26T02:36:12.007941+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:12.007941+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11870, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.014057+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:12.014057+0000', 'last_peered': '2024-04-26T02:36:12.014057+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:12.014057+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "40'11987", 'reported_seq': 15364, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:26.637953+0000', 'last_change': '2024-04-26T02:36:10.613092+0000', 'last_active': '2024-04-26T02:36:26.637953+0000', 'last_peered': '2024-04-26T02:36:26.637953+0000', 'last_clean': '2024-04-26T02:23:51.430840+0000', 'last_became_active': '2024-04-26T02:24:59.611357+0000', 'last_became_peered': '2024-04-26T02:24:59.611357+0000', 'last_unstale': '2024-04-26T02:36:26.637953+0000', 'last_undegraded': '2024-04-26T02:24:59.294664+0000', 'last_fullsized': '2024-04-26T02:24:59.276384+0000', 'mapping_epoch': 31, 'log_start': "28'1900", 'ondisk_log_start': "28'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 500, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11985, 'num_write_kb': 47948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1645, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:33.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.005339+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:12.005339+0000', 'last_peered': '2024-04-26T02:36:12.005339+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:12.005339+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11964, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.009493+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:12.009493+0000', 'last_peered': '2024-04-26T02:36:12.009493+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:12.009493+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.011774+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:12.011774+0000', 'last_peered': '2024-04-26T02:36:12.011774+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:12.011774+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12009, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.003303+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:12.003303+0000', 'last_peered': '2024-04-26T02:36:12.003303+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:12.003303+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12381, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:12.006820+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:12.006820+0000', 'last_peered': '2024-04-26T02:36:12.006820+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:12.006820+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:33.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12040, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:31.102424+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:31.102424+0000', 'last_peered': '2024-04-26T02:36:31.102424+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:31.102424+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:33.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:34.325 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18157/112 objects degraded (16211.607%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2702540800,"bytes_avail":383844515840,"bytes_total":386547056640,"degraded_objects":18157,"degraded_total":112,"degraded_ratio":162.11607142857142,"recovering_objects_per_sec":64,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":518,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-26T02:35:36.763192+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:34.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:34.648 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:34.665 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-04-26T02:36:32.784534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639200,"kb_used_data":1644,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848160,"statfs":{"total":386547056640,"available":383844515840,"internally_reserved":0,"allocated":1683456,"data_stored":750557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003050"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":11770,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009476+0000","last_change":"2024-04-26T02:24:59.581976+0000","last_active":"2024-04-26T02:36:12.009476+0000","last_peered":"2024-04-26T02:36:12.009476+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:12.009476+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.004037+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:12.004037+0000","last_peered":"2024-04-26T02:36:12.004037+0000","last_clean":"2024-04-26T02:36:12.004037+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:12.004037+0000","last_undegraded":"2024-04-26T02:36:12.004037+0000","last_fullsized":"2024-04-26T02:36:12.004037+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.277796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15792,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.099001+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:12.099001+0000","last_peered":"2024-04-26T02:36:12.099001+0000","last_clean":"2024-04-26T02:36:12.099001+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:12.099001+0000","last_undegraded":"2024-04-26T02:36:12.099001+0000","last_fullsized":"2024-04-26T02:36:12.099001+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15517,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003438+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:12.003438+0000","last_peered":"2024-04-26T02:36:12.003438+0000","last_clean":"2024-04-26T02:36:12.003438+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:12.003438+0000","last_undegraded":"2024-04-26T02:36:12.003438+0000","last_fullsized":"2024-04-26T02:36:12.003438+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.007941+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:12.007941+0000","last_peered":"2024-04-26T02:36:12.007941+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:12.007941+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.014057+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:12.014057+0000","last_peered":"2024-04-26T02:36:12.014057+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:12.014057+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16024,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-26T02:36:31.095323+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:31.095323+0000","last_peered":"2024-04-26T02:36:31.095323+0000","last_clean":"2024-04-26T02:36:31.095323+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:31.095323+0000","last_undegraded":"2024-04-26T02:36:31.095323+0000","last_fullsized":"2024-04-26T02:36:31.095323+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":15364,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:26.637953+0000","last_change":"2024-04-26T02:36:10.613092+0000","last_active":"2024-04-26T02:36:26.637953+0000","last_peered":"2024-04-26T02:36:26.637953+0000","last_clean":"2024-04-26T02:23:51.430840+0000","last_became_active":"2024-04-26T02:24:59.611357+0000","last_became_peered":"2024-04-26T02:24:59.611357+0000","last_unstale":"2024-04-26T02:36:26.637953+0000","last_undegraded":"2024-04-26T02:24:59.294664+0000","last_fullsized":"2024-04-26T02:24:59.276384+0000","mapping_epoch":31,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15840,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.009273+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:12.009273+0000","last_peered":"2024-04-26T02:36:12.009273+0000","last_clean":"2024-04-26T02:36:12.009273+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:12.009273+0000","last_undegraded":"2024-04-26T02:36:12.009273+0000","last_fullsized":"2024-04-26T02:36:12.009273+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:12.458945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.005339+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:12.005339+0000","last_peered":"2024-04-26T02:36:12.005339+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:12.005339+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.009493+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:12.009493+0000","last_peered":"2024-04-26T02:36:12.009493+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:12.009493+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.011774+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:12.011774+0000","last_peered":"2024-04-26T02:36:12.011774+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:12.011774+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12009,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.003303+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:12.003303+0000","last_peered":"2024-04-26T02:36:12.003303+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:12.003303+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12381,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:12.006820+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:12.006820+0000","last_peered":"2024-04-26T02:36:12.006820+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:12.006820+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16077,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-26T02:36:12.003739+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:12.003739+0000","last_peered":"2024-04-26T02:36:12.003739+0000","last_clean":"2024-04-26T02:36:12.003739+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:12.003739+0000","last_undegraded":"2024-04-26T02:36:12.003739+0000","last_fullsized":"2024-04-26T02:36:12.003739+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:15.199567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12040,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:31.102424+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:31.102424+0000","last_peered":"2024-04-26T02:36:31.102424+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:31.102424+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14612,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685948,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685892,"statfs":{"total":96636764160,"available":95934353408,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689596,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682244,"statfs":{"total":96636764160,"available":95930617856,"internally_reserved":0,"allocated":372736,"data_stored":140709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569448,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802392,"statfs":{"total":96636764160,"available":96053649408,"internally_reserved":0,"allocated":557056,"data_stored":325029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694208,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677632,"statfs":{"total":96636764160,"available":95925895168,"internally_reserved":0,"allocated":380928,"data_stored":144110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:37.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:38.011 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:38.011 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:38.035 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-04-26T02:36:36.786281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647576,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839784,"statfs":{"total":386547056640,"available":383835938816,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003337"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":12953,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:35.043625+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:35.043625+0000","last_peered":"2024-04-26T02:36:35.043625+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:35.043625+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:38.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:38.360 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:38.377 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-04-26T02:36:36.786281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647576,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839784,"statfs":{"total":386547056640,"available":383835938816,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003337"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":12953,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:35.043625+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:35.043625+0000","last_peered":"2024-04-26T02:36:35.043625+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:35.043625+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:38.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:38.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 12953, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:35.043625+0000', 'last_change': '2024-04-26T02:36:30.312929+0000', 'last_active': '2024-04-26T02:36:35.043625+0000', 'last_peered': '2024-04-26T02:36:35.043625+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:35.043625+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1338, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 679, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1338}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11844, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.121734+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:32.121734+0000', 'last_peered': '2024-04-26T02:36:32.121734+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:32.121734+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11874, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.122225+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:32.122225+0000', 'last_peered': '2024-04-26T02:36:32.122225+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:32.122225+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11754, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113663+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:32.113663+0000', 'last_peered': '2024-04-26T02:36:32.113663+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:32.113663+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11968, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124518+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:32.124518+0000', 'last_peered': '2024-04-26T02:36:32.124518+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:32.124518+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11843, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124602+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:32.124602+0000', 'last_peered': '2024-04-26T02:36:32.124602+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:32.124602+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:38.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12013, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113279+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:32.113279+0000', 'last_peered': '2024-04-26T02:36:32.113279+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:32.113279+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:38.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:38.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12385, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124008+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:32.124008+0000', 'last_peered': '2024-04-26T02:36:32.124008+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:32.124008+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:38.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:38.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12042, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.114553+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:32.114553+0000', 'last_peered': '2024-04-26T02:36:32.114553+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:32.114553+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:38.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:38.735 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:38.735 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17068/112 objects degraded (15239.286%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":795,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711117824,"bytes_avail":383835938816,"bytes_total":386547056640,"degraded_objects":17068,"degraded_total":112,"degraded_ratio":152.39285714285714,"recovering_objects_per_sec":136,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1089,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-26T02:36:34.785550+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:38.754 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:38.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:39.057 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:39.058 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:39.073 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-04-26T02:36:38.787050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647576,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839784,"statfs":{"total":386547056640,"available":383835938816,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003305"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":12953,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:35.043625+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:35.043625+0000","last_peered":"2024-04-26T02:36:35.043625+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:35.043625+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15701,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019019,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:42.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:42.380 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:42.397 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-26T02:36:40.787553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651728,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835632,"statfs":{"total":386547056640,"available":383831687168,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.700259"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":13851,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:40.045641+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:40.045641+0000","last_peered":"2024-04-26T02:36:40.045641+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:40.045641+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581944,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789896,"statfs":{"total":96636764160,"available":96040853504,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:42.398 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:42.698 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:42.716 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-26T02:36:40.787553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651728,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835632,"statfs":{"total":386547056640,"available":383831687168,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.700259"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":13851,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:40.045641+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:40.045641+0000","last_peered":"2024-04-26T02:36:40.045641+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:40.045641+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581944,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789896,"statfs":{"total":96636764160,"available":96040853504,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 13851, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:40.045641+0000', 'last_change': '2024-04-26T02:36:30.312929+0000', 'last_active': '2024-04-26T02:36:40.045641+0000', 'last_peered': '2024-04-26T02:36:40.045641+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:40.045641+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 889, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1128, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -889}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11844, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.121734+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:32.121734+0000', 'last_peered': '2024-04-26T02:36:32.121734+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:32.121734+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11874, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.122225+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:32.122225+0000', 'last_peered': '2024-04-26T02:36:32.122225+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:32.122225+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11754, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113663+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:32.113663+0000', 'last_peered': '2024-04-26T02:36:32.113663+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:32.113663+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:42.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11968, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124518+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:32.124518+0000', 'last_peered': '2024-04-26T02:36:32.124518+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:32.124518+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11843, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124602+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:32.124602+0000', 'last_peered': '2024-04-26T02:36:32.124602+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:32.124602+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12013, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113279+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:32.113279+0000', 'last_peered': '2024-04-26T02:36:32.113279+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:32.113279+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12385, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124008+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:32.124008+0000', 'last_peered': '2024-04-26T02:36:32.124008+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:32.124008+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:42.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12042, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.114553+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:32.114553+0000', 'last_peered': '2024-04-26T02:36:32.114553+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:32.114553+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:42.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:43.088 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16619/112 objects degraded (14838.393%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":799,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715369472,"bytes_avail":383831687168,"bytes_total":386547056640,"degraded_objects":16619,"degraded_total":112,"degraded_ratio":148.38392857142858,"recovering_objects_per_sec":158,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1538,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-26T02:36:34.785550+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:43.107 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:43.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:43.402 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:43.403 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:43.419 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-04-26T02:36:42.788476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651728,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835632,"statfs":{"total":386547056640,"available":383831687168,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.677642"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":13851,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:40.045641+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:40.045641+0000","last_peered":"2024-04-26T02:36:40.045641+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:40.045641+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581944,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789896,"statfs":{"total":96636764160,"available":96040853504,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:46.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:46.744 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:46.744 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:46.763 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-04-26T02:36:44.789161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639248,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848112,"statfs":{"total":386547056640,"available":383844466688,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004627"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":13851,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:40.045641+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:40.045641+0000","last_peered":"2024-04-26T02:36:40.045641+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:40.045641+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16026,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113234+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113234+0000","last_peered":"2024-04-26T02:36:32.113234+0000","last_clean":"2024-04-26T02:36:32.113234+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113234+0000","last_undegraded":"2024-04-26T02:36:32.113234+0000","last_fullsized":"2024-04-26T02:36:32.113234+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:35.566769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16150,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569464,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802376,"statfs":{"total":96636764160,"available":96053633024,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:46.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:47.061 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:47.062 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:47.081 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-04-26T02:36:46.789862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16716,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639248,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848112,"statfs":{"total":386547056640,"available":383844466688,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004448"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":14983,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:45.043355+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:45.043355+0000","last_peered":"2024-04-26T02:36:45.043355+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:45.043355+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16027,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113307+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113307+0000","last_peered":"2024-04-26T02:36:32.113307+0000","last_clean":"2024-04-26T02:36:32.113307+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113307+0000","last_undegraded":"2024-04-26T02:36:32.113307+0000","last_fullsized":"2024-04-26T02:36:32.113307+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:46.667140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16716,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569464,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802376,"statfs":{"total":96636764160,"available":96053633024,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:47.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-04-26T02:36:47.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "40'11524", 'reported_seq': 14983, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:45.043355+0000', 'last_change': '2024-04-26T02:36:30.312929+0000', 'last_active': '2024-04-26T02:36:45.043355+0000', 'last_peered': '2024-04-26T02:36:45.043355+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.578364+0000', 'last_became_peered': '2024-04-26T02:24:59.578364+0000', 'last_unstale': '2024-04-26T02:36:45.043355+0000', 'last_undegraded': '2024-04-26T02:24:59.303290+0000', 'last_fullsized': '2024-04-26T02:24:59.228306+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11524, 'num_write_kb': 46096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1694, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:47.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11844, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.121734+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:32.121734+0000', 'last_peered': '2024-04-26T02:36:32.121734+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:32.121734+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11874, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.122225+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:32.122225+0000', 'last_peered': '2024-04-26T02:36:32.122225+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:32.122225+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11754, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113663+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:32.113663+0000', 'last_peered': '2024-04-26T02:36:32.113663+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:32.113663+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11968, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124518+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:32.124518+0000', 'last_peered': '2024-04-26T02:36:32.124518+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:32.124518+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11843, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124602+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:32.124602+0000', 'last_peered': '2024-04-26T02:36:32.124602+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:32.124602+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:47.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12013, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.113279+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:32.113279+0000', 'last_peered': '2024-04-26T02:36:32.113279+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:32.113279+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:47.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:47.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12385, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.124008+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:32.124008+0000', 'last_peered': '2024-04-26T02:36:32.124008+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:32.124008+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:47.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:47.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12042, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.114553+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:32.114553+0000', 'last_peered': '2024-04-26T02:36:32.114553+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:32.114553+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:47.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:47.443 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:47.444 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16053/112 objects degraded (14333.036%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2702589952,"bytes_avail":383844466688,"bytes_total":386547056640,"degraded_objects":16053,"degraded_total":112,"degraded_ratio":143.33035714285714,"recovering_objects_per_sec":133,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-26T02:36:34.785550+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:47.463 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:47.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:47.802 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:47.803 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:47.822 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-04-26T02:36:46.789862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16716,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639248,"kb_used_data":1700,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848112,"statfs":{"total":386547056640,"available":383844466688,"internally_reserved":0,"allocated":1740800,"data_stored":774280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004448"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":14983,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:45.043355+0000","last_change":"2024-04-26T02:36:30.312929+0000","last_active":"2024-04-26T02:36:45.043355+0000","last_peered":"2024-04-26T02:36:45.043355+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.578364+0000","last_became_peered":"2024-04-26T02:24:59.578364+0000","last_unstale":"2024-04-26T02:36:45.043355+0000","last_undegraded":"2024-04-26T02:24:59.303290+0000","last_fullsized":"2024-04-26T02:24:59.228306+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15415,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.120398+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:32.120398+0000","last_peered":"2024-04-26T02:36:32.120398+0000","last_clean":"2024-04-26T02:36:32.120398+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:32.120398+0000","last_undegraded":"2024-04-26T02:36:32.120398+0000","last_fullsized":"2024-04-26T02:36:32.120398+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:34.297432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15796,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113513+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:32.113513+0000","last_peered":"2024-04-26T02:36:32.113513+0000","last_clean":"2024-04-26T02:36:32.113513+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:32.113513+0000","last_undegraded":"2024-04-26T02:36:32.113513+0000","last_fullsized":"2024-04-26T02:36:32.113513+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:33.570233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15521,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.118833+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:32.118833+0000","last_peered":"2024-04-26T02:36:32.118833+0000","last_clean":"2024-04-26T02:36:32.118833+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:32.118833+0000","last_undegraded":"2024-04-26T02:36:32.118833+0000","last_fullsized":"2024-04-26T02:36:32.118833+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11844,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.121734+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:32.121734+0000","last_peered":"2024-04-26T02:36:32.121734+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:32.121734+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.122225+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:32.122225+0000","last_peered":"2024-04-26T02:36:32.122225+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:32.122225+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16027,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113307+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113307+0000","last_peered":"2024-04-26T02:36:32.113307+0000","last_clean":"2024-04-26T02:36:32.113307+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113307+0000","last_undegraded":"2024-04-26T02:36:32.113307+0000","last_fullsized":"2024-04-26T02:36:32.113307+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:46.667140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12261,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.214032+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:32.214032+0000","last_peered":"2024-04-26T02:36:32.214032+0000","last_clean":"2024-04-26T02:36:32.214032+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:32.214032+0000","last_undegraded":"2024-04-26T02:36:32.214032+0000","last_fullsized":"2024-04-26T02:36:32.214032+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:37.279936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15844,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.122057+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:32.122057+0000","last_peered":"2024-04-26T02:36:32.122057+0000","last_clean":"2024-04-26T02:36:32.122057+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:32.122057+0000","last_undegraded":"2024-04-26T02:36:32.122057+0000","last_fullsized":"2024-04-26T02:36:32.122057+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:32.595066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113663+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:32.113663+0000","last_peered":"2024-04-26T02:36:32.113663+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:32.113663+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11968,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124518+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:32.124518+0000","last_peered":"2024-04-26T02:36:32.124518+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:32.124518+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11843,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124602+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:32.124602+0000","last_peered":"2024-04-26T02:36:32.124602+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:32.124602+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.113279+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:32.113279+0000","last_peered":"2024-04-26T02:36:32.113279+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:32.113279+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.124008+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:32.124008+0000","last_peered":"2024-04-26T02:36:32.124008+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:32.124008+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16081,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.119010+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:32.119010+0000","last_peered":"2024-04-26T02:36:32.119010+0000","last_clean":"2024-04-26T02:36:32.119010+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:32.119010+0000","last_undegraded":"2024-04-26T02:36:32.119010+0000","last_fullsized":"2024-04-26T02:36:32.119010+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:36.276128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16716,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685960,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685880,"statfs":{"total":96636764160,"available":95934341120,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689608,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682232,"statfs":{"total":96636764160,"available":95930605568,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019021,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569464,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802376,"statfs":{"total":96636764160,"available":96053633024,"internally_reserved":0,"allocated":573440,"data_stored":331810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:50.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:51.135 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:51.135 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:51.155 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-04-26T02:36:50.791041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643436,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843924,"statfs":{"total":386547056640,"available":383840178176,"internally_reserved":0,"allocated":1765376,"data_stored":779523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002564"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15417,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302936+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:48.302936+0000","last_peered":"2024-04-26T02:36:48.302936+0000","last_clean":"2024-04-26T02:36:48.302936+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:48.302936+0000","last_undegraded":"2024-04-26T02:36:48.302936+0000","last_fullsized":"2024-04-26T02:36:48.302936+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15798,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.283161+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:48.283161+0000","last_peered":"2024-04-26T02:36:48.283161+0000","last_clean":"2024-04-26T02:36:48.283161+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:48.283161+0000","last_undegraded":"2024-04-26T02:36:48.283161+0000","last_fullsized":"2024-04-26T02:36:48.283161+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.301949+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:48.301949+0000","last_peered":"2024-04-26T02:36:48.301949+0000","last_clean":"2024-04-26T02:36:48.301949+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:48.301949+0000","last_undegraded":"2024-04-26T02:36:48.301949+0000","last_fullsized":"2024-04-26T02:36:48.301949+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16027,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113307+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113307+0000","last_peered":"2024-04-26T02:36:32.113307+0000","last_clean":"2024-04-26T02:36:32.113307+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113307+0000","last_undegraded":"2024-04-26T02:36:32.113307+0000","last_fullsized":"2024-04-26T02:36:32.113307+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:46.667140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12263,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302132+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:48.302132+0000","last_peered":"2024-04-26T02:36:48.302132+0000","last_clean":"2024-04-26T02:36:48.302132+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:48.302132+0000","last_undegraded":"2024-04-26T02:36:48.302132+0000","last_fullsized":"2024-04-26T02:36:48.302132+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11756,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.283534+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:48.283534+0000","last_peered":"2024-04-26T02:36:48.283534+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:48.283534+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.282246+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:48.282246+0000","last_peered":"2024-04-26T02:36:48.282246+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:48.282246+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16083,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302299+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:48.302299+0000","last_peered":"2024-04-26T02:36:48.302299+0000","last_clean":"2024-04-26T02:36:48.302299+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:48.302299+0000","last_undegraded":"2024-04-26T02:36:48.302299+0000","last_fullsized":"2024-04-26T02:36:48.302299+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689616,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682224,"statfs":{"total":96636764160,"available":95930597376,"internally_reserved":0,"allocated":397312,"data_stored":150854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573628,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798212,"statfs":{"total":96636764160,"available":96049369088,"internally_reserved":0,"allocated":573440,"data_stored":326982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:51.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:51.454 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:51.455 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-04-26T02:36:50.791041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643436,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843924,"statfs":{"total":386547056640,"available":383840178176,"internally_reserved":0,"allocated":1765376,"data_stored":779523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002564"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15417,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302936+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:48.302936+0000","last_peered":"2024-04-26T02:36:48.302936+0000","last_clean":"2024-04-26T02:36:48.302936+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:48.302936+0000","last_undegraded":"2024-04-26T02:36:48.302936+0000","last_fullsized":"2024-04-26T02:36:48.302936+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15798,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.283161+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:48.283161+0000","last_peered":"2024-04-26T02:36:48.283161+0000","last_clean":"2024-04-26T02:36:48.283161+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:48.283161+0000","last_undegraded":"2024-04-26T02:36:48.283161+0000","last_fullsized":"2024-04-26T02:36:48.283161+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.301949+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:48.301949+0000","last_peered":"2024-04-26T02:36:48.301949+0000","last_clean":"2024-04-26T02:36:48.301949+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:48.301949+0000","last_undegraded":"2024-04-26T02:36:48.301949+0000","last_fullsized":"2024-04-26T02:36:48.301949+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16027,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113307+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113307+0000","last_peered":"2024-04-26T02:36:32.113307+0000","last_clean":"2024-04-26T02:36:32.113307+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113307+0000","last_undegraded":"2024-04-26T02:36:32.113307+0000","last_fullsized":"2024-04-26T02:36:32.113307+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:46.667140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12263,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302132+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:48.302132+0000","last_peered":"2024-04-26T02:36:48.302132+0000","last_clean":"2024-04-26T02:36:48.302132+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:48.302132+0000","last_undegraded":"2024-04-26T02:36:48.302132+0000","last_fullsized":"2024-04-26T02:36:48.302132+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11756,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.283534+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:48.283534+0000","last_peered":"2024-04-26T02:36:48.283534+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:48.283534+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.282246+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:48.282246+0000","last_peered":"2024-04-26T02:36:48.282246+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:48.282246+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16083,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302299+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:48.302299+0000","last_peered":"2024-04-26T02:36:48.302299+0000","last_clean":"2024-04-26T02:36:48.302299+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:48.302299+0000","last_undegraded":"2024-04-26T02:36:48.302299+0000","last_fullsized":"2024-04-26T02:36:48.302299+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689616,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682224,"statfs":{"total":96636764160,"available":95930597376,"internally_reserved":0,"allocated":397312,"data_stored":150854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573628,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798212,"statfs":{"total":96636764160,"available":96049369088,"internally_reserved":0,"allocated":573440,"data_stored":326982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.315767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:49.315767+0000', 'last_peered': '2024-04-26T02:36:49.315767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:49.315767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11878, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.318696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:49.318696+0000', 'last_peered': '2024-04-26T02:36:49.318696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:49.318696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11756, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:48.283534+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:48.283534+0000', 'last_peered': '2024-04-26T02:36:48.283534+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:48.283534+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:51.475 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11972, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319948+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:49.319948+0000', 'last_peered': '2024-04-26T02:36:49.319948+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:49.319948+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11847, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.399706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:49.399706+0000', 'last_peered': '2024-04-26T02:36:49.399706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:49.399706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12015, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:48.282246+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:48.282246+0000', 'last_peered': '2024-04-26T02:36:48.282246+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:48.282246+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12389, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319391+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:49.319391+0000', 'last_peered': '2024-04-26T02:36:49.319391+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:49.319391+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:51.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12042, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:32.114553+0000', 'last_change': '2024-04-26T02:24:59.638491+0000', 'last_active': '2024-04-26T02:36:32.114553+0000', 'last_peered': '2024-04-26T02:36:32.114553+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:32.114553+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:51.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:51.875 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:51.875 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15730/112 objects degraded (14044.643%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":8}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2706878464,"bytes_avail":383840178176,"bytes_total":386547056640,"degraded_objects":15730,"degraded_total":112,"degraded_ratio":140.44642857142858,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":889,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-26T02:36:34.785550+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:51.894 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:51.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:52.193 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:52.209 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-04-26T02:36:50.791041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643436,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843924,"statfs":{"total":386547056640,"available":383840178176,"internally_reserved":0,"allocated":1765376,"data_stored":779523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002564"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15417,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302936+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:48.302936+0000","last_peered":"2024-04-26T02:36:48.302936+0000","last_clean":"2024-04-26T02:36:48.302936+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:48.302936+0000","last_undegraded":"2024-04-26T02:36:48.302936+0000","last_fullsized":"2024-04-26T02:36:48.302936+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15798,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.283161+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:48.283161+0000","last_peered":"2024-04-26T02:36:48.283161+0000","last_clean":"2024-04-26T02:36:48.283161+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:48.283161+0000","last_undegraded":"2024-04-26T02:36:48.283161+0000","last_fullsized":"2024-04-26T02:36:48.283161+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15524,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.301949+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:48.301949+0000","last_peered":"2024-04-26T02:36:48.301949+0000","last_clean":"2024-04-26T02:36:48.301949+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:48.301949+0000","last_undegraded":"2024-04-26T02:36:48.301949+0000","last_fullsized":"2024-04-26T02:36:48.301949+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16027,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-26T02:36:32.113307+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:32.113307+0000","last_peered":"2024-04-26T02:36:32.113307+0000","last_clean":"2024-04-26T02:36:32.113307+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:32.113307+0000","last_undegraded":"2024-04-26T02:36:32.113307+0000","last_fullsized":"2024-04-26T02:36:32.113307+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:46.667140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12263,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302132+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:48.302132+0000","last_peered":"2024-04-26T02:36:48.302132+0000","last_clean":"2024-04-26T02:36:48.302132+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:48.302132+0000","last_undegraded":"2024-04-26T02:36:48.302132+0000","last_fullsized":"2024-04-26T02:36:48.302132+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11756,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.283534+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:48.283534+0000","last_peered":"2024-04-26T02:36:48.283534+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:48.283534+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:48.282246+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:48.282246+0000","last_peered":"2024-04-26T02:36:48.282246+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:48.282246+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16083,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-26T02:36:48.302299+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:48.302299+0000","last_peered":"2024-04-26T02:36:48.302299+0000","last_clean":"2024-04-26T02:36:48.302299+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:48.302299+0000","last_undegraded":"2024-04-26T02:36:48.302299+0000","last_fullsized":"2024-04-26T02:36:48.302299+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12042,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:32.114553+0000","last_change":"2024-04-26T02:24:59.638491+0000","last_active":"2024-04-26T02:36:32.114553+0000","last_peered":"2024-04-26T02:36:32.114553+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:32.114553+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1985}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17039,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689616,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682224,"statfs":{"total":96636764160,"available":95930597376,"internally_reserved":0,"allocated":397312,"data_stored":150854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573628,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798212,"statfs":{"total":96636764160,"available":96049369088,"internally_reserved":0,"allocated":573440,"data_stored":326982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694216,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677624,"statfs":{"total":96636764160,"available":95925886976,"internally_reserved":0,"allocated":389120,"data_stored":147490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:55.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:55.529 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:55.529 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:55.548 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-04-26T02:36:54.792574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647628,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839732,"statfs":{"total":386547056640,"available":383835885568,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002711"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12979,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:51.643148+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:51.643148+0000","last_peered":"2024-04-26T02:36:51.643148+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:51.643148+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:55.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:55.906 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:55.907 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:55.924 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-04-26T02:36:54.792574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647628,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839732,"statfs":{"total":386547056640,"available":383835885568,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002711"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12979,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:51.643148+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:51.643148+0000","last_peered":"2024-04-26T02:36:51.643148+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:51.643148+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.315767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:49.315767+0000', 'last_peered': '2024-04-26T02:36:49.315767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:49.315767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11878, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.318696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:49.318696+0000', 'last_peered': '2024-04-26T02:36:49.318696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:49.318696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11758, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.311313+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:49.311313+0000', 'last_peered': '2024-04-26T02:36:49.311313+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:49.311313+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:55.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11972, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319948+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:49.319948+0000', 'last_peered': '2024-04-26T02:36:49.319948+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:49.319948+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11847, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.399706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:49.399706+0000', 'last_peered': '2024-04-26T02:36:49.399706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:49.399706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12017, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.310366+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:49.310366+0000', 'last_peered': '2024-04-26T02:36:49.310366+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:49.310366+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12389, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319391+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:49.319391+0000', 'last_peered': '2024-04-26T02:36:49.319391+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:49.319391+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:36:55.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 12979, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:51.643148+0000', 'last_change': '2024-04-26T02:36:47.324320+0000', 'last_active': '2024-04-26T02:36:51.643148+0000', 'last_peered': '2024-04-26T02:36:51.643148+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:51.643148+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1523, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 545, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:36:55.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:36:56.299 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:56.299 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15264/112 objects degraded (13628.571%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711171072,"bytes_avail":383835885568,"bytes_total":386547056640,"degraded_objects":15264,"degraded_total":112,"degraded_ratio":136.28571428571428,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":789,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-26T02:36:34.785550+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:36:56.317 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:36:56.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:36:56.635 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:36:56.635 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:36:56.654 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-04-26T02:36:54.792574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647628,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839732,"statfs":{"total":386547056640,"available":383835885568,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002711"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15848,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318566+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318566+0000","last_peered":"2024-04-26T02:36:49.318566+0000","last_clean":"2024-04-26T02:36:49.318566+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318566+0000","last_undegraded":"2024-04-26T02:36:49.318566+0000","last_fullsized":"2024-04-26T02:36:49.318566+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.702078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11972,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319948+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.319948+0000","last_peered":"2024-04-26T02:36:49.319948+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.319948+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12979,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:51.643148+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:51.643148+0000","last_peered":"2024-04-26T02:36:51.643148+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:51.643148+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":545,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17505,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:36:59.656 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:00.016 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:00.017 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:00.035 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-04-26T02:36:58.794159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647628,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839732,"statfs":{"total":386547056640,"available":383835885568,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003872"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":13879,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:56.640665+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:56.640665+0000","last_peered":"2024-04-26T02:36:56.640665+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:56.640665+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:00.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:00.335 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:00.353 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-04-26T02:36:58.794159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647628,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839732,"statfs":{"total":386547056640,"available":383835885568,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003872"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":13879,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:56.640665+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:56.640665+0000","last_peered":"2024-04-26T02:36:56.640665+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:56.640665+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577792,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794048,"statfs":{"total":96636764160,"available":96045105152,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.315767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:49.315767+0000', 'last_peered': '2024-04-26T02:36:49.315767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:49.315767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11878, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.318696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:49.318696+0000', 'last_peered': '2024-04-26T02:36:49.318696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:49.318696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11758, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.311313+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:49.311313+0000', 'last_peered': '2024-04-26T02:36:49.311313+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:49.311313+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:00.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11973, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.321718+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:49.321718+0000', 'last_peered': '2024-04-26T02:36:49.321718+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:49.321718+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11847, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.399706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:49.399706+0000', 'last_peered': '2024-04-26T02:36:49.399706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:49.399706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12017, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.310366+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:49.310366+0000', 'last_peered': '2024-04-26T02:36:49.310366+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:49.310366+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12389, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319391+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:49.319391+0000', 'last_peered': '2024-04-26T02:36:49.319391+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:49.319391+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:37:00.359 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 13879, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:56.640665+0000', 'last_change': '2024-04-26T02:36:47.324320+0000', 'last_active': '2024-04-26T02:36:56.640665+0000', 'last_peered': '2024-04-26T02:36:56.640665+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:36:56.640665+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 995, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1071}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:37:00.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:00.731 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:00.732 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14814/112 objects degraded (13226.786%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":817,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711171072,"bytes_avail":383835885568,"bytes_total":386547056640,"degraded_objects":14814,"degraded_total":112,"degraded_ratio":132.26785714285714,"recovering_objects_per_sec":123,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1239,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:00.751 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:00.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:01.043 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:01.060 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-04-26T02:37:00.794856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651788,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835572,"statfs":{"total":386547056640,"available":383831625728,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.486154"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":13879,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:56.640665+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:36:56.640665+0000","last_peered":"2024-04-26T02:36:56.640665+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:36:56.640665+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":995,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1071}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17955,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581952,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789888,"statfs":{"total":96636764160,"available":96040845312,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:04.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:04.377 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:04.395 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-04-26T02:37:02.795349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651788,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835572,"statfs":{"total":386547056640,"available":383831625728,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004307"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581952,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789888,"statfs":{"total":96636764160,"available":96040845312,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:04.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:04.712 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:04.712 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:04.731 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-04-26T02:37:02.795349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651788,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835572,"statfs":{"total":386547056640,"available":383831625728,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1415,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004307"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581952,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789888,"statfs":{"total":96636764160,"available":96040845312,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.315767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:49.315767+0000', 'last_peered': '2024-04-26T02:36:49.315767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:49.315767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11878, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.318696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:49.318696+0000', 'last_peered': '2024-04-26T02:36:49.318696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:49.318696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11758, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.311313+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:36:49.311313+0000', 'last_peered': '2024-04-26T02:36:49.311313+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:36:49.311313+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:04.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11973, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.321718+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:49.321718+0000', 'last_peered': '2024-04-26T02:36:49.321718+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:49.321718+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11847, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.399706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:49.399706+0000', 'last_peered': '2024-04-26T02:36:49.399706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:49.399706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12017, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.310366+0000', 'last_change': '2024-04-26T02:24:59.619226+0000', 'last_active': '2024-04-26T02:36:49.310366+0000', 'last_peered': '2024-04-26T02:36:49.310366+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:36:49.310366+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12389, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319391+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:49.319391+0000', 'last_peered': '2024-04-26T02:36:49.319391+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:49.319391+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:37:04.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 14877, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:01.644876+0000', 'last_change': '2024-04-26T02:36:47.324320+0000', 'last_active': '2024-04-26T02:37:01.644876+0000', 'last_peered': '2024-04-26T02:37:01.644876+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:37:01.644876+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 574, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1494, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:37:04.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:05.104 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:05.105 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14315/112 objects degraded (12781.250%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2702651392,"bytes_avail":383844405248,"bytes_total":386547056640,"degraded_objects":14315,"degraded_total":112,"degraded_ratio":127.8125,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":949,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:05.125 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:05.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:05.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:05.436 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:05.453 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-04-26T02:37:04.796001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639308,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848052,"statfs":{"total":386547056640,"available":383844405248,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004167"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.311418+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:36:49.311418+0000","last_peered":"2024-04-26T02:36:49.311418+0000","last_clean":"2024-04-26T02:36:49.311418+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:36:49.311418+0000","last_undegraded":"2024-04-26T02:36:49.311418+0000","last_fullsized":"2024-04-26T02:36:49.311418+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:51.673715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569472,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802368,"statfs":{"total":96636764160,"available":96053624832,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:08.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:08.785 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:08.786 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:08.805 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-04-26T02:37:06.875974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639308,"kb_used_data":1756,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848052,"statfs":{"total":386547056640,"available":383844405248,"internally_reserved":0,"allocated":1798144,"data_stored":792916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.081815"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15800,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.310962+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:36:49.310962+0000","last_peered":"2024-04-26T02:36:49.310962+0000","last_clean":"2024-04-26T02:36:49.310962+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:36:49.310962+0000","last_undegraded":"2024-04-26T02:36:49.310962+0000","last_fullsized":"2024-04-26T02:36:49.310962+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:50.403028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16033,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-26T02:37:05.870098+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:05.870098+0000","last_peered":"2024-04-26T02:37:05.870098+0000","last_clean":"2024-04-26T02:37:05.870098+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:05.870098+0000","last_undegraded":"2024-04-26T02:37:05.870098+0000","last_fullsized":"2024-04-26T02:37:05.870098+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.311313+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:36:49.311313+0000","last_peered":"2024-04-26T02:36:49.311313+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:36:49.311313+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.310366+0000","last_change":"2024-04-26T02:24:59.619226+0000","last_active":"2024-04-26T02:36:49.310366+0000","last_peered":"2024-04-26T02:36:49.310366+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:36:49.310366+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2044}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18454,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689624,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682216,"statfs":{"total":96636764160,"available":95930589184,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569472,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802368,"statfs":{"total":96636764160,"available":96053624832,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:08.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:09.126 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:09.126 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:09.145 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-04-26T02:37:08.797283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18635,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639328,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848032,"statfs":{"total":386547056640,"available":383844384768,"internally_reserved":0,"allocated":1814528,"data_stored":799549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002426"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16033,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-26T02:37:05.870098+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:05.870098+0000","last_peered":"2024-04-26T02:37:05.870098+0000","last_clean":"2024-04-26T02:37:05.870098+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:05.870098+0000","last_undegraded":"2024-04-26T02:37:05.870098+0000","last_fullsized":"2024-04-26T02:37:05.870098+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12385,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:08.489780+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:08.489780+0000","last_peered":"2024-04-26T02:37:08.489780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:08.489780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18635,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569472,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802368,"statfs":{"total":96636764160,"available":96053624832,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.315767+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:36:49.315767+0000', 'last_peered': '2024-04-26T02:36:49.315767+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:36:49.315767+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11878, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.318696+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:36:49.318696+0000', 'last_peered': '2024-04-26T02:36:49.318696+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:36:49.318696+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:09.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11762, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888444+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:06.888444+0000', 'last_peered': '2024-04-26T02:37:06.888444+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:06.888444+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11973, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.321718+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:36:49.321718+0000', 'last_peered': '2024-04-26T02:36:49.321718+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:36:49.321718+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11847, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.399706+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:36:49.399706+0000', 'last_peered': '2024-04-26T02:36:49.399706+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:36:49.399706+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12385, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:08.489780+0000', 'last_change': '2024-04-26T02:37:05.606551+0000', 'last_active': '2024-04-26T02:37:08.489780+0000', 'last_peered': '2024-04-26T02:37:08.489780+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:37:08.489780+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1865, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 248, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:09.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12389, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:36:49.319391+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:36:49.319391+0000', 'last_peered': '2024-04-26T02:36:49.319391+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:36:49.319391+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:09.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-04-26T02:37:09.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "40'11795", 'reported_seq': 14877, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:01.644876+0000', 'last_change': '2024-04-26T02:36:47.324320+0000', 'last_active': '2024-04-26T02:37:01.644876+0000', 'last_peered': '2024-04-26T02:37:01.644876+0000', 'last_clean': '2024-04-26T02:23:50.353302+0000', 'last_became_active': '2024-04-26T02:24:59.606906+0000', 'last_became_peered': '2024-04-26T02:24:59.606906+0000', 'last_unstale': '2024-04-26T02:37:01.644876+0000', 'last_undegraded': '2024-04-26T02:24:59.267130+0000', 'last_fullsized': '2024-04-26T02:24:59.232969+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 574, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11793, 'num_write_kb': 47180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1494, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T02:37:09.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:09.558 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14134/112 objects degraded (12619.643%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2702671872,"bytes_avail":383844384768,"bytes_total":386547056640,"degraded_objects":14134,"degraded_total":112,"degraded_ratio":126.19642857142857,"recovering_objects_per_sec":84,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:09.579 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:09.581 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:09.886 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:09.886 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:09.904 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-04-26T02:37:08.797283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18635,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639328,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374848032,"statfs":{"total":386547056640,"available":383844384768,"internally_reserved":0,"allocated":1814528,"data_stored":799549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002426"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15641,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.399864+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:36:49.399864+0000","last_peered":"2024-04-26T02:36:49.399864+0000","last_clean":"2024-04-26T02:36:49.399864+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:36:49.399864+0000","last_undegraded":"2024-04-26T02:36:49.399864+0000","last_fullsized":"2024-04-26T02:36:49.399864+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319242+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:36:49.319242+0000","last_peered":"2024-04-26T02:36:49.319242+0000","last_clean":"2024-04-26T02:36:49.319242+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:36:49.319242+0000","last_undegraded":"2024-04-26T02:36:49.319242+0000","last_fullsized":"2024-04-26T02:36:49.319242+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:49.341458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15526,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318773+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:36:49.318773+0000","last_peered":"2024-04-26T02:36:49.318773+0000","last_clean":"2024-04-26T02:36:49.318773+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:36:49.318773+0000","last_undegraded":"2024-04-26T02:36:49.318773+0000","last_fullsized":"2024-04-26T02:36:49.318773+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.315767+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:36:49.315767+0000","last_peered":"2024-04-26T02:36:49.315767+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:36:49.315767+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11878,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.318696+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:36:49.318696+0000","last_peered":"2024-04-26T02:36:49.318696+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:36:49.318696+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16033,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-26T02:37:05.870098+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:05.870098+0000","last_peered":"2024-04-26T02:37:05.870098+0000","last_clean":"2024-04-26T02:37:05.870098+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:05.870098+0000","last_undegraded":"2024-04-26T02:37:05.870098+0000","last_fullsized":"2024-04-26T02:37:05.870098+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12265,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319474+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:36:49.319474+0000","last_peered":"2024-04-26T02:36:49.319474+0000","last_clean":"2024-04-26T02:36:49.319474+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:36:49.319474+0000","last_undegraded":"2024-04-26T02:36:49.319474+0000","last_fullsized":"2024-04-26T02:36:49.319474+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15849,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.318618+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:36:49.318618+0000","last_peered":"2024-04-26T02:36:49.318618+0000","last_clean":"2024-04-26T02:36:49.318618+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:36:49.318618+0000","last_undegraded":"2024-04-26T02:36:49.318618+0000","last_fullsized":"2024-04-26T02:36:49.318618+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:55.672323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11973,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.321718+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:36:49.321718+0000","last_peered":"2024-04-26T02:36:49.321718+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:36:49.321718+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11847,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.399706+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:36:49.399706+0000","last_peered":"2024-04-26T02:36:49.399706+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:36:49.399706+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12385,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:08.489780+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:08.489780+0000","last_peered":"2024-04-26T02:37:08.489780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:08.489780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12389,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:36:49.319391+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:36:49.319391+0000","last_peered":"2024-04-26T02:36:49.319391+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:36:49.319391+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16085,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-26T02:36:49.319381+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:36:49.319381+0000","last_peered":"2024-04-26T02:36:49.319381+0000","last_clean":"2024-04-26T02:36:49.319381+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:36:49.319381+0000","last_undegraded":"2024-04-26T02:36:49.319381+0000","last_fullsized":"2024-04-26T02:36:49.319381+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:52.403152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":14877,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:01.644876+0000","last_change":"2024-04-26T02:36:47.324320+0000","last_active":"2024-04-26T02:37:01.644876+0000","last_peered":"2024-04-26T02:37:01.644876+0000","last_clean":"2024-04-26T02:23:50.353302+0000","last_became_active":"2024-04-26T02:24:59.606906+0000","last_became_peered":"2024-04-26T02:24:59.606906+0000","last_unstale":"2024-04-26T02:37:01.644876+0000","last_undegraded":"2024-04-26T02:24:59.267130+0000","last_fullsized":"2024-04-26T02:24:59.232969+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18635,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685976,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685864,"statfs":{"total":96636764160,"available":95934324736,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019025,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569472,"kb_used_data":568,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802368,"statfs":{"total":96636764160,"available":96053624832,"internally_reserved":0,"allocated":581632,"data_stored":330325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694236,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677604,"statfs":{"total":96636764160,"available":95925866496,"internally_reserved":0,"allocated":405504,"data_stored":154197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:12.906 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:13.231 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:13.231 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:13.250 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-04-26T02:37:12.798823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643540,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843820,"statfs":{"total":386547056640,"available":383840071680,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002822"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11851,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.886931+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.886931+0000","last_peered":"2024-04-26T02:37:06.886931+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.886931+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12385,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:08.489780+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:08.489780+0000","last_peered":"2024-04-26T02:37:08.489780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:08.489780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:13.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:13.577 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:13.578 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:13.596 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-04-26T02:37:12.798823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643540,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843820,"statfs":{"total":386547056640,"available":383840071680,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002822"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11851,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.886931+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.886931+0000","last_peered":"2024-04-26T02:37:06.886931+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.886931+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12385,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:08.489780+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:08.489780+0000","last_peered":"2024-04-26T02:37:08.489780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:08.489780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.879938+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:37:06.879938+0000', 'last_peered': '2024-04-26T02:37:06.879938+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:06.879938+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11882, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.881096+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:06.881096+0000', 'last_peered': '2024-04-26T02:37:06.881096+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:06.881096+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11762, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888444+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:06.888444+0000', 'last_peered': '2024-04-26T02:37:06.888444+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:06.888444+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:13.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11977, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.884457+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:06.884457+0000', 'last_peered': '2024-04-26T02:37:06.884457+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:06.884457+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11851, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.886931+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:06.886931+0000', 'last_peered': '2024-04-26T02:37:06.886931+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:06.886931+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 12385, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:08.489780+0000', 'last_change': '2024-04-26T02:37:05.606551+0000', 'last_active': '2024-04-26T02:37:08.489780+0000', 'last_peered': '2024-04-26T02:37:08.489780+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:37:08.489780+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1865, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 248, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:13.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12393, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.883517+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:06.883517+0000', 'last_peered': '2024-04-26T02:37:06.883517+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:06.883517+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:13.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:13.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:13.979 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13560/112 objects degraded (12107.143%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2706984960,"bytes_avail":383840071680,"bytes_total":386547056640,"degraded_objects":13560,"degraded_total":112,"degraded_ratio":121.07142857142857,"recovering_objects_per_sec":94,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":755,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:13.997 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:13.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:14.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:14.304 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:14.322 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-04-26T02:37:12.798823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643540,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843820,"statfs":{"total":386547056640,"available":383840071680,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002822"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11851,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.886931+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.886931+0000","last_peered":"2024-04-26T02:37:06.886931+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.886931+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12385,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:08.489780+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:08.489780+0000","last_peered":"2024-04-26T02:37:08.489780+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:08.489780+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1864}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19209,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215269,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:17.324 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:17.641 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:17.658 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-26T02:37:16.800519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647696,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839664,"statfs":{"total":386547056640,"available":383835815936,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004099"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":13181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:13.491649+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:13.491649+0000","last_peered":"2024-04-26T02:37:13.491649+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:13.491649+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577804,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794036,"statfs":{"total":96636764160,"available":96045092864,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:17.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:17.999 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:18.000 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:18.017 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-26T02:37:16.800519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647696,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839664,"statfs":{"total":386547056640,"available":383835815936,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004099"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":13181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:13.491649+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:13.491649+0000","last_peered":"2024-04-26T02:37:13.491649+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:13.491649+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577804,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794036,"statfs":{"total":96636764160,"available":96045092864,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:18.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:18.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.879938+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:37:06.879938+0000', 'last_peered': '2024-04-26T02:37:06.879938+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:06.879938+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:18.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:18.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11882, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.881096+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:06.881096+0000', 'last_peered': '2024-04-26T02:37:06.881096+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:06.881096+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11762, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888444+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:06.888444+0000', 'last_peered': '2024-04-26T02:37:06.888444+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:06.888444+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11977, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.884457+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:06.884457+0000', 'last_peered': '2024-04-26T02:37:06.884457+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:06.884457+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888619+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:06.888619+0000', 'last_peered': '2024-04-26T02:37:06.888619+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:06.888619+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 13181, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:13.491649+0000', 'last_change': '2024-04-26T02:37:05.606551+0000', 'last_active': '2024-04-26T02:37:13.491649+0000', 'last_peered': '2024-04-26T02:37:13.491649+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:37:13.491649+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1467, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 646, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:18.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12393, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.883517+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:06.883517+0000', 'last_peered': '2024-04-26T02:37:06.883517+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:06.883517+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:18.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:18.376 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:18.377 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13162/112 objects degraded (11751.786%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":834,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711240704,"bytes_avail":383835815936,"bytes_total":386547056640,"degraded_objects":13162,"degraded_total":112,"degraded_ratio":117.51785714285714,"recovering_objects_per_sec":115,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:18.398 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:18.398 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:18.699 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:18.700 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:18.718 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-04-26T02:37:16.800519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647696,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839664,"statfs":{"total":386547056640,"available":383835815936,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004099"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15804,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888083+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888083+0000","last_peered":"2024-04-26T02:37:06.888083+0000","last_clean":"2024-04-26T02:37:06.888083+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888083+0000","last_undegraded":"2024-04-26T02:37:06.888083+0000","last_fullsized":"2024-04-26T02:37:06.888083+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:08.152762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":13181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:13.491649+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:13.491649+0000","last_peered":"2024-04-26T02:37:13.491649+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:13.491649+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577804,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794036,"statfs":{"total":96636764160,"available":96045092864,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:21.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:22.046 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:22.065 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-04-26T02:37:20.801934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651856,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835504,"statfs":{"total":386547056640,"available":383831556096,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004651"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":14053,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:18.492638+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:18.492638+0000","last_peered":"2024-04-26T02:37:18.492638+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:18.492638+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1030}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581964,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789876,"statfs":{"total":96636764160,"available":96040833024,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:22.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:22.371 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:22.371 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:22.389 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-04-26T02:37:20.801934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651856,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835504,"statfs":{"total":386547056640,"available":383831556096,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004651"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":14053,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:18.492638+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:18.492638+0000","last_peered":"2024-04-26T02:37:18.492638+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:18.492638+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1030}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581964,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789876,"statfs":{"total":96636764160,"available":96040833024,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.879938+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:37:06.879938+0000', 'last_peered': '2024-04-26T02:37:06.879938+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:06.879938+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:22.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11882, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.881096+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:06.881096+0000', 'last_peered': '2024-04-26T02:37:06.881096+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:06.881096+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11762, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888444+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:06.888444+0000', 'last_peered': '2024-04-26T02:37:06.888444+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:06.888444+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11977, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.884457+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:06.884457+0000', 'last_peered': '2024-04-26T02:37:06.884457+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:06.884457+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888619+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:06.888619+0000', 'last_peered': '2024-04-26T02:37:06.888619+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:06.888619+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:22.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 14053, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:18.492638+0000', 'last_change': '2024-04-26T02:37:05.606551+0000', 'last_active': '2024-04-26T02:37:18.492638+0000', 'last_peered': '2024-04-26T02:37:18.492638+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:37:18.492638+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1082, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1030}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:22.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:22.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12393, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.883517+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:06.883517+0000', 'last_peered': '2024-04-26T02:37:06.883517+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:06.883517+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:22.392 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:22.792 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:22.792 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12726/112 objects degraded (11362.500%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715500544,"bytes_avail":383831556096,"bytes_total":386547056640,"degraded_objects":12726,"degraded_total":112,"degraded_ratio":113.625,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1408,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:22.812 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:22.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:23.140 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:23.140 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:23.156 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-04-26T02:37:22.802788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651856,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835504,"statfs":{"total":386547056640,"available":383831556096,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":834,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004833"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":14053,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:18.492638+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:18.492638+0000","last_peered":"2024-04-26T02:37:18.492638+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:18.492638+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1030}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20043,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019028,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581964,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789876,"statfs":{"total":96636764160,"available":96040833024,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:26.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:26.478 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:26.478 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:26.495 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-26T02:37:24.803307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639376,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374847984,"statfs":{"total":386547056640,"available":383844335616,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004483"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":15181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:23.496004+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:23.496004+0000","last_peered":"2024-04-26T02:37:23.496004+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:23.496004+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569484,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802356,"statfs":{"total":96636764160,"available":96053612544,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:26.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:26.790 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:26.790 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:26.809 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-26T02:37:24.803307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639376,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374847984,"statfs":{"total":386547056640,"available":383844335616,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004483"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":15181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:23.496004+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:23.496004+0000","last_peered":"2024-04-26T02:37:23.496004+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:23.496004+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569484,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802356,"statfs":{"total":96636764160,"available":96053612544,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.879938+0000', 'last_change': '2024-04-26T02:24:59.587018+0000', 'last_active': '2024-04-26T02:37:06.879938+0000', 'last_peered': '2024-04-26T02:37:06.879938+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:06.879938+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1931, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1927}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11882, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.881096+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:06.881096+0000', 'last_peered': '2024-04-26T02:37:06.881096+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:06.881096+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11762, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888444+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:06.888444+0000', 'last_peered': '2024-04-26T02:37:06.888444+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:06.888444+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:26.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11977, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.884457+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:06.884457+0000', 'last_peered': '2024-04-26T02:37:06.884457+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:06.884457+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11852, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.888619+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:06.888619+0000', 'last_peered': '2024-04-26T02:37:06.888619+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:06.888619+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "40'11778", 'reported_seq': 15181, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:23.496004+0000', 'last_change': '2024-04-26T02:37:05.606551+0000', 'last_active': '2024-04-26T02:37:23.496004+0000', 'last_peered': '2024-04-26T02:37:23.496004+0000', 'last_clean': '2024-04-26T02:23:43.663091+0000', 'last_became_active': '2024-04-26T02:24:59.606038+0000', 'last_became_peered': '2024-04-26T02:24:59.606038+0000', 'last_unstale': '2024-04-26T02:37:23.496004+0000', 'last_undegraded': '2024-04-26T02:24:59.262145+0000', 'last_fullsized': '2024-04-26T02:24:59.229886+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 467, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 47111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1646, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:26.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12393, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:06.883517+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:06.883517+0000', 'last_peered': '2024-04-26T02:37:06.883517+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:06.883517+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:26.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:27.177 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12162/112 objects degraded (10858.929%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2702721024,"bytes_avail":383844335616,"bytes_total":386547056640,"degraded_objects":12162,"degraded_total":112,"degraded_ratio":108.58928571428571,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1000,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:27.195 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:27.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:27.510 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:27.526 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-04-26T02:37:26.804021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2639376,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":2637489,"kb_avail":374847984,"statfs":{"total":386547056640,"available":383844335616,"internally_reserved":0,"allocated":1859584,"data_stored":815352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2700789299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004511"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15645,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.886735+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:06.886735+0000","last_peered":"2024-04-26T02:37:06.886735+0000","last_clean":"2024-04-26T02:37:06.886735+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:06.886735+0000","last_undegraded":"2024-04-26T02:37:06.886735+0000","last_fullsized":"2024-04-26T02:37:06.886735+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15423,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880994+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:06.880994+0000","last_peered":"2024-04-26T02:37:06.880994+0000","last_clean":"2024-04-26T02:37:06.880994+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:06.880994+0000","last_undegraded":"2024-04-26T02:37:06.880994+0000","last_fullsized":"2024-04-26T02:37:06.880994+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:10.576212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15805,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.888226+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:06.888226+0000","last_peered":"2024-04-26T02:37:06.888226+0000","last_clean":"2024-04-26T02:37:06.888226+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:06.888226+0000","last_undegraded":"2024-04-26T02:37:06.888226+0000","last_fullsized":"2024-04-26T02:37:06.888226+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:19.079084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15530,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.881728+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:06.881728+0000","last_peered":"2024-04-26T02:37:06.881728+0000","last_clean":"2024-04-26T02:37:06.881728+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:06.881728+0000","last_undegraded":"2024-04-26T02:37:06.881728+0000","last_fullsized":"2024-04-26T02:37:06.881728+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.879938+0000","last_change":"2024-04-26T02:24:59.587018+0000","last_active":"2024-04-26T02:37:06.879938+0000","last_peered":"2024-04-26T02:37:06.879938+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:06.879938+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11882,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.881096+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:06.881096+0000","last_peered":"2024-04-26T02:37:06.881096+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:06.881096+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12269,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880389+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:06.880389+0000","last_peered":"2024-04-26T02:37:06.880389+0000","last_clean":"2024-04-26T02:37:06.880389+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:06.880389+0000","last_undegraded":"2024-04-26T02:37:06.880389+0000","last_fullsized":"2024-04-26T02:37:06.880389+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15853,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880896+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:06.880896+0000","last_peered":"2024-04-26T02:37:06.880896+0000","last_clean":"2024-04-26T02:37:06.880896+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:06.880896+0000","last_undegraded":"2024-04-26T02:37:06.880896+0000","last_fullsized":"2024-04-26T02:37:06.880896+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.747318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11762,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888444+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:06.888444+0000","last_peered":"2024-04-26T02:37:06.888444+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:06.888444+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11977,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.884457+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:06.884457+0000","last_peered":"2024-04-26T02:37:06.884457+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:06.884457+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11852,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.888619+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:06.888619+0000","last_peered":"2024-04-26T02:37:06.888619+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:06.888619+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":15181,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:23.496004+0000","last_change":"2024-04-26T02:37:05.606551+0000","last_active":"2024-04-26T02:37:23.496004+0000","last_peered":"2024-04-26T02:37:23.496004+0000","last_clean":"2024-04-26T02:23:43.663091+0000","last_became_active":"2024-04-26T02:24:59.606038+0000","last_became_peered":"2024-04-26T02:24:59.606038+0000","last_unstale":"2024-04-26T02:37:23.496004+0000","last_undegraded":"2024-04-26T02:24:59.262145+0000","last_fullsized":"2024-04-26T02:24:59.229886+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12393,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:06.883517+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:06.883517+0000","last_peered":"2024-04-26T02:37:06.883517+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:06.883517+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16089,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.880455+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:06.880455+0000","last_peered":"2024-04-26T02:37:06.880455+0000","last_clean":"2024-04-26T02:37:06.880455+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:06.880455+0000","last_undegraded":"2024-04-26T02:37:06.880455+0000","last_fullsized":"2024-04-26T02:37:06.880455+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.510236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12044,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.995570+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:06.995570+0000","last_peered":"2024-04-26T02:37:06.995570+0000","last_clean":"2024-04-26T02:37:06.995570+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:06.995570+0000","last_undegraded":"2024-04-26T02:37:06.995570+0000","last_fullsized":"2024-04-26T02:37:06.995570+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:12.534512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685996,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685844,"statfs":{"total":96636764160,"available":95934304256,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689644,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682196,"statfs":{"total":96636764160,"available":95930568704,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569484,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":568891,"kb_avail":93802356,"statfs":{"total":96636764160,"available":96053612544,"internally_reserved":0,"allocated":593920,"data_stored":332862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582545103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:30.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:30.858 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:30.858 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:30.877 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-04-26T02:37:30.805099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643556,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843804,"statfs":{"total":386547056640,"available":383840055296,"internally_reserved":0,"allocated":1884160,"data_stored":820299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-683,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002311"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15807,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T02:37:27.949428+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:27.949428+0000","last_peered":"2024-04-26T02:37:27.949428+0000","last_clean":"2024-04-26T02:37:27.949428+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:27.949428+0000","last_undegraded":"2024-04-26T02:37:27.949428+0000","last_fullsized":"2024-04-26T02:37:27.949428+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11764,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:27.949761+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:27.949761+0000","last_peered":"2024-04-26T02:37:27.949761+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:27.949761+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":430080,"data_stored":164120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":327960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:30.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:31.198 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:31.199 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:31.217 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-04-26T02:37:30.805099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643556,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843804,"statfs":{"total":386547056640,"available":383840055296,"internally_reserved":0,"allocated":1884160,"data_stored":820299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-683,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002311"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15807,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T02:37:27.949428+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:27.949428+0000","last_peered":"2024-04-26T02:37:27.949428+0000","last_clean":"2024-04-26T02:37:27.949428+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:27.949428+0000","last_undegraded":"2024-04-26T02:37:27.949428+0000","last_fullsized":"2024-04-26T02:37:27.949428+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11764,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:27.949761+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:27.949761+0000","last_peered":"2024-04-26T02:37:27.949761+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:27.949761+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":430080,"data_stored":164120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":327960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 12521, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:30.052505+0000', 'last_change': '2024-04-26T02:37:27.184690+0000', 'last_active': '2024-04-26T02:37:30.052505+0000', 'last_peered': '2024-04-26T02:37:30.052505+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:30.052505+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 416, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11886, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949591+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:28.949591+0000', 'last_peered': '2024-04-26T02:37:28.949591+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:28.949591+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11764, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:27.949761+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:27.949761+0000', 'last_peered': '2024-04-26T02:37:27.949761+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:27.949761+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:31.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11981, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949546+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:28.949546+0000', 'last_peered': '2024-04-26T02:37:28.949546+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:28.949546+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:31.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:31.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11856, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.952786+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:28.952786+0000', 'last_peered': '2024-04-26T02:37:28.952786+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:28.952786+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:31.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:31.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12398, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951204+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:28.951204+0000', 'last_peered': '2024-04-26T02:37:28.951204+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:28.951204+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:31.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:31.585 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:31.585 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11363/112 objects degraded (10145.536%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":848,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2707001344,"bytes_avail":383840055296,"bytes_total":386547056640,"degraded_objects":11363,"degraded_total":112,"degraded_ratio":101.45535714285714},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:31.608 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:37:31.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:31.938 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:31.938 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:31.957 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-04-26T02:37:30.805099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643556,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":2641649,"kb_avail":374843804,"statfs":{"total":386547056640,"available":383840055296,"internally_reserved":0,"allocated":1884160,"data_stored":820299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2705049139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-683,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002311"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15807,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T02:37:27.949428+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:27.949428+0000","last_peered":"2024-04-26T02:37:27.949428+0000","last_clean":"2024-04-26T02:37:27.949428+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:27.949428+0000","last_undegraded":"2024-04-26T02:37:27.949428+0000","last_fullsized":"2024-04-26T02:37:27.949428+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16035,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-26T02:37:06.883108+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:06.883108+0000","last_peered":"2024-04-26T02:37:06.883108+0000","last_clean":"2024-04-26T02:37:06.883108+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:06.883108+0000","last_undegraded":"2024-04-26T02:37:06.883108+0000","last_fullsized":"2024-04-26T02:37:06.883108+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:07.761086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11764,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:27.949761+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:27.949761+0000","last_peered":"2024-04-26T02:37:27.949761+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:27.949761+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689648,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682192,"statfs":{"total":96636764160,"available":95930564608,"internally_reserved":0,"allocated":430080,"data_stored":164120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573648,"kb_used_data":580,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798192,"statfs":{"total":96636764160,"available":96049348608,"internally_reserved":0,"allocated":593920,"data_stored":327960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694252,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677588,"statfs":{"total":96636764160,"available":95925850112,"internally_reserved":0,"allocated":421888,"data_stored":160830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:34.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:35.281 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:35.281 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:35.299 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-26T02:37:34.806693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002672"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:35.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:35.611 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:35.612 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:35.632 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-26T02:37:34.806693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002672"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 12521, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:30.052505+0000', 'last_change': '2024-04-26T02:37:27.184690+0000', 'last_active': '2024-04-26T02:37:30.052505+0000', 'last_peered': '2024-04-26T02:37:30.052505+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:30.052505+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 416, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11886, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949591+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:28.949591+0000', 'last_peered': '2024-04-26T02:37:28.949591+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:28.949591+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11766, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.948209+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:28.948209+0000', 'last_peered': '2024-04-26T02:37:28.948209+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:28.948209+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:35.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11981, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949546+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:28.949546+0000', 'last_peered': '2024-04-26T02:37:28.949546+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:28.949546+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:35.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:35.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11856, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.952786+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:28.952786+0000', 'last_peered': '2024-04-26T02:37:28.952786+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:28.952786+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:35.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:35.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12398, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951204+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:28.951204+0000', 'last_peered': '2024-04-26T02:37:28.951204+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:28.951204+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:35.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:36.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:36.044 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11363/112 objects degraded (10145.536%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":852,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711285760,"bytes_avail":383835770880,"bytes_total":386547056640,"degraded_objects":11363,"degraded_total":112,"degraded_ratio":101.45535714285714},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:36.063 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:37:36.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:36.364 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:36.365 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:36.382 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-26T02:37:34.806693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002672"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":12521,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:30.052505+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:30.052505+0000","last_peered":"2024-04-26T02:37:30.052505+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:30.052505+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19360,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:39.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:39.706 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:39.706 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:39.726 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-26T02:37:38.808159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-819,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003625"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":13377,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:35.057593+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:35.057593+0000","last_peered":"2024-04-26T02:37:35.057593+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:35.057593+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:39.728 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:40.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:40.038 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:40.055 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-26T02:37:38.808159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-819,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003625"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":13377,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:35.057593+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:35.057593+0000","last_peered":"2024-04-26T02:37:35.057593+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:35.057593+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 13377, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:35.057593+0000', 'last_change': '2024-04-26T02:37:27.184690+0000', 'last_active': '2024-04-26T02:37:35.057593+0000', 'last_peered': '2024-04-26T02:37:35.057593+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:35.057593+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1171, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 844, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11886, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949591+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:28.949591+0000', 'last_peered': '2024-04-26T02:37:28.949591+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:28.949591+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11766, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.948209+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:28.948209+0000', 'last_peered': '2024-04-26T02:37:28.948209+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:28.948209+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11981, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949546+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:28.949546+0000', 'last_peered': '2024-04-26T02:37:28.949546+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:28.949546+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:40.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11856, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.952786+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:28.952786+0000', 'last_peered': '2024-04-26T02:37:28.952786+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:28.952786+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:40.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:40.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12398, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951204+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:28.951204+0000', 'last_peered': '2024-04-26T02:37:28.951204+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:28.951204+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:40.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:40.440 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:40.440 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10935/112 objects degraded (9763.393%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":856,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2711285760,"bytes_avail":383835770880,"bytes_total":386547056640,"degraded_objects":10935,"degraded_total":112,"degraded_ratio":97.633928571428569},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:40.460 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:37:40.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:40.762 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:40.762 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:40.780 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-04-26T02:37:38.808159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2647740,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2645809,"kb_avail":374839620,"statfs":{"total":386547056640,"available":383835770880,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2709308979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-819,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003625"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":13377,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:35.057593+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:35.057593+0000","last_peered":"2024-04-26T02:37:35.057593+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:35.057593+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11886,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949591+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.949591+0000","last_peered":"2024-04-26T02:37:28.949591+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.949591+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16093,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946903+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946903+0000","last_peered":"2024-04-26T02:37:28.946903+0000","last_clean":"2024-04-26T02:37:28.946903+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946903+0000","last_undegraded":"2024-04-26T02:37:28.946903+0000","last_fullsized":"2024-04-26T02:37:28.946903+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19788,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:43.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:44.120 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:44.121 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:44.141 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-04-26T02:37:42.809275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651900,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835460,"statfs":{"total":386547056640,"available":383831511040,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004175"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":14389,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:40.057573+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:40.057573+0000","last_peered":"2024-04-26T02:37:40.057573+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:40.057573+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581972,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789868,"statfs":{"total":96636764160,"available":96040824832,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:44.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:44.486 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:44.486 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:44.505 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-04-26T02:37:42.809275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2651900,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2649969,"kb_avail":374835460,"statfs":{"total":386547056640,"available":383831511040,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2713568819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004175"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":14389,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:40.057573+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:40.057573+0000","last_peered":"2024-04-26T02:37:40.057573+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:40.057573+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581972,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93789868,"statfs":{"total":96636764160,"available":96040824832,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:44.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:44.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 14389, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:40.057573+0000', 'last_change': '2024-04-26T02:37:27.184690+0000', 'last_active': '2024-04-26T02:37:40.057573+0000', 'last_peered': '2024-04-26T02:37:40.057573+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:40.057573+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 665, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1350, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -663}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11887, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951125+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:28.951125+0000', 'last_peered': '2024-04-26T02:37:28.951125+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:28.951125+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 11766, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.948209+0000', 'last_change': '2024-04-26T02:24:59.662606+0000', 'last_active': '2024-04-26T02:37:28.948209+0000', 'last_peered': '2024-04-26T02:37:28.948209+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:28.948209+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11981, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949546+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:28.949546+0000', 'last_peered': '2024-04-26T02:37:28.949546+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:28.949546+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11856, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.952786+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:28.952786+0000', 'last_peered': '2024-04-26T02:37:28.952786+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:28.952786+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:44.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12398, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951204+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:28.951204+0000', 'last_peered': '2024-04-26T02:37:28.951204+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:28.951204+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:44.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:44.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:44.897 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10429/112 objects degraded (9311.607%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":861,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2715545600,"bytes_avail":383831511040,"bytes_total":386547056640,"degraded_objects":10429,"degraded_total":112,"degraded_ratio":93.116071428571431,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:44.917 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:44.917 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:45.217 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:45.237 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-04-26T02:37:44.809899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657468,"kb_used_data":1872,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829892,"statfs":{"total":386547056640,"available":383825809408,"internally_reserved":0,"allocated":1916928,"data_stored":833396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004085"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":14389,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:40.057573+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:40.057573+0000","last_peered":"2024-04-26T02:37:40.057573+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:40.057573+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16039,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952029+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:28.952029+0000","last_peered":"2024-04-26T02:37:28.952029+0000","last_clean":"2024-04-26T02:37:28.952029+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:28.952029+0000","last_undegraded":"2024-04-26T02:37:28.952029+0000","last_fullsized":"2024-04-26T02:37:28.952029+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:29.616942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20294,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587540,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784300,"statfs":{"total":96636764160,"available":96035123200,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215275,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694264,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677576,"statfs":{"total":96636764160,"available":95925837824,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:48.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:48.575 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:48.592 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-04-26T02:37:47.063980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657476,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829884,"statfs":{"total":386547056640,"available":383825801216,"internally_reserved":0,"allocated":1925120,"data_stored":836649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.255821"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15809,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.947881+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:28.947881+0000","last_peered":"2024-04-26T02:37:28.947881+0000","last_clean":"2024-04-26T02:37:28.947881+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:28.947881+0000","last_undegraded":"2024-04-26T02:37:28.947881+0000","last_fullsized":"2024-04-26T02:37:28.947881+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.011270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15713,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:45.062880+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:45.062880+0000","last_peered":"2024-04-26T02:37:45.062880+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:45.062880+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16041,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-26T02:37:46.045758+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:46.045758+0000","last_peered":"2024-04-26T02:37:46.045758+0000","last_clean":"2024-04-26T02:37:46.045758+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:46.045758+0000","last_undegraded":"2024-04-26T02:37:46.045758+0000","last_fullsized":"2024-04-26T02:37:46.045758+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":11766,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.948209+0000","last_change":"2024-04-26T02:24:59.662606+0000","last_active":"2024-04-26T02:37:28.948209+0000","last_peered":"2024-04-26T02:37:28.948209+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:28.948209+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689656,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682184,"statfs":{"total":96636764160,"available":95930556416,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587540,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784300,"statfs":{"total":96636764160,"available":96035123200,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694272,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677568,"statfs":{"total":96636764160,"available":95925829632,"internally_reserved":0,"allocated":446464,"data_stored":170642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:48.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:48.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:48.897 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:48.916 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-04-26T02:37:48.811052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21407,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657492,"kb_used_data":1896,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829868,"statfs":{"total":386547056640,"available":383825784832,"internally_reserved":0,"allocated":1941504,"data_stored":843134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002279"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15713,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:45.062880+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:45.062880+0000","last_peered":"2024-04-26T02:37:45.062880+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:45.062880+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16041,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-26T02:37:46.045758+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:46.045758+0000","last_peered":"2024-04-26T02:37:46.045758+0000","last_clean":"2024-04-26T02:37:46.045758+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:46.045758+0000","last_undegraded":"2024-04-26T02:37:46.045758+0000","last_fullsized":"2024-04-26T02:37:46.045758+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":12673,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:48.496635+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:48.496635+0000","last_peered":"2024-04-26T02:37:48.496635+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:48.496635+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21407,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587540,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784300,"statfs":{"total":96636764160,"available":96035123200,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694272,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677568,"statfs":{"total":96636764160,"available":95925829632,"internally_reserved":0,"allocated":446464,"data_stored":170642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:48.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-04-26T02:37:48.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "40'11595", 'reported_seq': 15713, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:45.062880+0000', 'last_change': '2024-04-26T02:37:27.184690+0000', 'last_active': '2024-04-26T02:37:45.062880+0000', 'last_peered': '2024-04-26T02:37:45.062880+0000', 'last_clean': '2024-04-26T02:23:50.231824+0000', 'last_became_active': '2024-04-26T02:24:59.579446+0000', 'last_became_peered': '2024-04-26T02:24:59.579446+0000', 'last_unstale': '2024-04-26T02:37:45.062880+0000', 'last_undegraded': '2024-04-26T02:24:59.316536+0000', 'last_fullsized': '2024-04-26T02:24:59.227341+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 46381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2012, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:48.917 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:48.917 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11887, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951125+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:28.951125+0000', 'last_peered': '2024-04-26T02:37:28.951125+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:28.951125+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 12673, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:48.496635+0000', 'last_change': '2024-04-26T02:37:45.081694+0000', 'last_active': '2024-04-26T02:37:48.496635+0000', 'last_peered': '2024-04-26T02:37:48.496635+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:48.496635+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1546, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 495, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11981, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.949546+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:28.949546+0000', 'last_peered': '2024-04-26T02:37:28.949546+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:28.949546+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11856, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.952786+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:28.952786+0000', 'last_peered': '2024-04-26T02:37:28.952786+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:28.952786+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:48.918 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12398, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:28.951204+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:28.951204+0000', 'last_peered': '2024-04-26T02:37:28.951204+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:28.951204+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:48.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:49.369 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:49.369 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9316/112 objects degraded (8317.857%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2721271808,"bytes_avail":383825784832,"bytes_total":386547056640,"degraded_objects":9316,"degraded_total":112,"degraded_ratio":83.178571428571431,"recovering_objects_per_sec":139,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1113,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:49.388 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:49.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:49.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:49.684 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:49.702 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-04-26T02:37:48.811052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21407,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657492,"kb_used_data":1896,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829868,"statfs":{"total":386547056640,"available":383825784832,"internally_reserved":0,"allocated":1941504,"data_stored":843134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002279"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.952438+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:28.952438+0000","last_peered":"2024-04-26T02:37:28.952438+0000","last_clean":"2024-04-26T02:37:28.952438+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:28.952438+0000","last_undegraded":"2024-04-26T02:37:28.952438+0000","last_fullsized":"2024-04-26T02:37:28.952438+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15427,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946923+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:28.946923+0000","last_peered":"2024-04-26T02:37:28.946923+0000","last_clean":"2024-04-26T02:37:28.946923+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:28.946923+0000","last_undegraded":"2024-04-26T02:37:28.946923+0000","last_fullsized":"2024-04-26T02:37:28.946923+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:31.637208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15534,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946032+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:28.946032+0000","last_peered":"2024-04-26T02:37:28.946032+0000","last_clean":"2024-04-26T02:37:28.946032+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:28.946032+0000","last_undegraded":"2024-04-26T02:37:28.946032+0000","last_fullsized":"2024-04-26T02:37:28.946032+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15713,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:45.062880+0000","last_change":"2024-04-26T02:37:27.184690+0000","last_active":"2024-04-26T02:37:45.062880+0000","last_peered":"2024-04-26T02:37:45.062880+0000","last_clean":"2024-04-26T02:23:50.231824+0000","last_became_active":"2024-04-26T02:24:59.579446+0000","last_became_peered":"2024-04-26T02:24:59.579446+0000","last_unstale":"2024-04-26T02:37:45.062880+0000","last_undegraded":"2024-04-26T02:24:59.316536+0000","last_fullsized":"2024-04-26T02:24:59.227341+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11887,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951125+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:28.951125+0000","last_peered":"2024-04-26T02:37:28.951125+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:28.951125+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16041,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-26T02:37:46.045758+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:46.045758+0000","last_peered":"2024-04-26T02:37:46.045758+0000","last_clean":"2024-04-26T02:37:46.045758+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:46.045758+0000","last_undegraded":"2024-04-26T02:37:46.045758+0000","last_fullsized":"2024-04-26T02:37:46.045758+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946568+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:28.946568+0000","last_peered":"2024-04-26T02:37:28.946568+0000","last_clean":"2024-04-26T02:37:28.946568+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:28.946568+0000","last_undegraded":"2024-04-26T02:37:28.946568+0000","last_fullsized":"2024-04-26T02:37:28.946568+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15857,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.950509+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:28.950509+0000","last_peered":"2024-04-26T02:37:28.950509+0000","last_clean":"2024-04-26T02:37:28.950509+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:28.950509+0000","last_undegraded":"2024-04-26T02:37:28.950509+0000","last_fullsized":"2024-04-26T02:37:28.950509+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:32.721735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":12673,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:48.496635+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:48.496635+0000","last_peered":"2024-04-26T02:37:48.496635+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:48.496635+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11981,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.949546+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:28.949546+0000","last_peered":"2024-04-26T02:37:28.949546+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:28.949546+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11856,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.952786+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:28.952786+0000","last_peered":"2024-04-26T02:37:28.952786+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:28.952786+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12015,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:29.048997+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:29.048997+0000","last_peered":"2024-04-26T02:37:29.048997+0000","last_clean":"2024-04-26T02:37:29.048997+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:29.048997+0000","last_undegraded":"2024-04-26T02:37:29.048997+0000","last_fullsized":"2024-04-26T02:37:29.048997+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12398,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:28.951204+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:28.951204+0000","last_peered":"2024-04-26T02:37:28.951204+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:28.951204+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16094,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946973+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:28.946973+0000","last_peered":"2024-04-26T02:37:28.946973+0000","last_clean":"2024-04-26T02:37:28.946973+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:28.946973+0000","last_undegraded":"2024-04-26T02:37:28.946973+0000","last_fullsized":"2024-04-26T02:37:28.946973+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:39.597274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12048,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T02:37:28.946773+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:28.946773+0000","last_peered":"2024-04-26T02:37:28.946773+0000","last_clean":"2024-04-26T02:37:28.946773+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:28.946773+0000","last_undegraded":"2024-04-26T02:37:28.946773+0000","last_fullsized":"2024-04-26T02:37:28.946773+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:33.628507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21407,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686008,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685832,"statfs":{"total":96636764160,"available":95934291968,"internally_reserved":0,"allocated":438272,"data_stored":167389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019034,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587540,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784300,"statfs":{"total":96636764160,"available":96035123200,"internally_reserved":0,"allocated":602112,"data_stored":331229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694272,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677568,"statfs":{"total":96636764160,"available":95925829632,"internally_reserved":0,"allocated":446464,"data_stored":170642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:52.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:53.029 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:53.050 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-26T02:37:52.812422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665852,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821508,"statfs":{"total":386547056640,"available":383817224192,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002523"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15653,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073444+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073444+0000","last_peered":"2024-04-26T02:37:47.073444+0000","last_clean":"2024-04-26T02:37:47.073444+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073444+0000","last_undegraded":"2024-04-26T02:37:47.073444+0000","last_fullsized":"2024-04-26T02:37:47.073444+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":12673,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:48.496635+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:48.496635+0000","last_peered":"2024-04-26T02:37:48.496635+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:48.496635+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:53.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:53.356 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:53.374 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-26T02:37:52.812422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665852,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821508,"statfs":{"total":386547056640,"available":383817224192,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002523"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15653,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073444+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073444+0000","last_peered":"2024-04-26T02:37:47.073444+0000","last_clean":"2024-04-26T02:37:47.073444+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073444+0000","last_undegraded":"2024-04-26T02:37:47.073444+0000","last_fullsized":"2024-04-26T02:37:47.073444+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":12673,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:48.496635+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:48.496635+0000","last_peered":"2024-04-26T02:37:48.496635+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:48.496635+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:53.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:53.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11891, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070757+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:47.070757+0000', 'last_peered': '2024-04-26T02:37:47.070757+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:47.070757+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:53.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:53.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 12673, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:48.496635+0000', 'last_change': '2024-04-26T02:37:45.081694+0000', 'last_active': '2024-04-26T02:37:48.496635+0000', 'last_peered': '2024-04-26T02:37:48.496635+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:48.496635+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1546, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 495, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:53.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:53.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11985, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073432+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:47.073432+0000', 'last_peered': '2024-04-26T02:37:47.073432+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:47.073432+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:53.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:53.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11860, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073673+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:47.073673+0000', 'last_peered': '2024-04-26T02:37:47.073673+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:47.073673+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:53.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:53.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12402, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070095+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:47.070095+0000', 'last_peered': '2024-04-26T02:37:47.070095+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:47.070095+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:53.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:53.768 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:53.769 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9313/112 objects degraded (8315.179%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2729832448,"bytes_avail":383817224192,"bytes_total":386547056640,"degraded_objects":9313,"degraded_total":112,"degraded_ratio":83.151785714285708,"recovering_objects_per_sec":139,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1116,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:53.787 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:53.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:54.093 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:54.093 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:54.111 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-04-26T02:37:52.812422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665852,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821508,"statfs":{"total":386547056640,"available":383817224192,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002523"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15653,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073444+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073444+0000","last_peered":"2024-04-26T02:37:47.073444+0000","last_clean":"2024-04-26T02:37:47.073444+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073444+0000","last_undegraded":"2024-04-26T02:37:47.073444+0000","last_fullsized":"2024-04-26T02:37:47.073444+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":12673,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:48.496635+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:48.496635+0000","last_peered":"2024-04-26T02:37:48.496635+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:48.496635+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21410,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595876,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775964,"statfs":{"total":96636764160,"available":96026587136,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215277,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:57.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:57.437 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:57.438 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:57.457 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-04-26T02:37:56.814047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21865,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003639"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":13583,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:53.493946+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:53.493946+0000","last_peered":"2024-04-26T02:37:53.493946+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:53.493946+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21865,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:57.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:58.599 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:58.600 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:58.621 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-04-26T02:37:56.814047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21865,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003639"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":13583,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:53.493946+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:53.493946+0000","last_peered":"2024-04-26T02:37:53.493946+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:53.493946+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21865,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:37:58.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11891, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070757+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:47.070757+0000', 'last_peered': '2024-04-26T02:37:47.070757+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:47.070757+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 13583, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:53.493946+0000', 'last_change': '2024-04-26T02:37:45.081694+0000', 'last_active': '2024-04-26T02:37:53.493946+0000', 'last_peered': '2024-04-26T02:37:53.493946+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:53.493946+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1091, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 950, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11985, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073432+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:47.073432+0000', 'last_peered': '2024-04-26T02:37:47.073432+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:47.073432+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11860, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073673+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:47.073673+0000', 'last_peered': '2024-04-26T02:37:47.073673+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:47.073673+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:37:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12402, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070095+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:47.070095+0000', 'last_peered': '2024-04-26T02:37:47.070095+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:47.070095+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:37:58.624 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:37:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:59.000 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8858/112 objects degraded (7908.929%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":875,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734084096,"bytes_avail":383812972544,"bytes_total":386547056640,"degraded_objects":8858,"degraded_total":112,"degraded_ratio":79.089285714285708,"recovering_objects_per_sec":90,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-26T02:36:56.793558+0000","services":{}},"progress_events":{}} 2024-04-26T02:37:59.018 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:37:59.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:37:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:37:59.346 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:37:59.362 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-04-26T02:37:58.814725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1427,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.750745"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":14619,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:58.497603+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:58.497603+0000","last_peered":"2024-04-26T02:37:58.497603+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:58.497603+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019036,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:02.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:02.682 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:02.682 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:02.700 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-04-26T02:38:00.815112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004060"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":14619,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:58.497603+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:58.497603+0000","last_peered":"2024-04-26T02:37:58.497603+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:58.497603+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:02.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:02.995 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:03.011 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-04-26T02:38:02.816067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":973,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004348"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":14619,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:58.497603+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:58.497603+0000","last_peered":"2024-04-26T02:37:58.497603+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:58.497603+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215279,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:03.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11891, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070757+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:37:47.070757+0000', 'last_peered': '2024-04-26T02:37:47.070757+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:37:47.070757+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 14619, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:58.497603+0000', 'last_change': '2024-04-26T02:37:45.081694+0000', 'last_active': '2024-04-26T02:37:58.497603+0000', 'last_peered': '2024-04-26T02:37:58.497603+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:37:58.497603+0000', 'last_undegraded': '2024-04-26T02:24:59.277203+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 573, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1468, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -571}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11985, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073432+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:37:47.073432+0000', 'last_peered': '2024-04-26T02:37:47.073432+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:37:47.073432+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 11860, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.073673+0000', 'last_change': '2024-04-26T02:24:59.612337+0000', 'last_active': '2024-04-26T02:37:47.073673+0000', 'last_peered': '2024-04-26T02:37:47.073673+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:37:47.073673+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1899, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:03.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12402, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:37:47.070095+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:37:47.070095+0000', 'last_peered': '2024-04-26T02:37:47.070095+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:37:47.070095+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:03.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:03.375 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:03.376 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8340/112 objects degraded (7446.429%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":879,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734084096,"bytes_avail":383812972544,"bytes_total":386547056640,"degraded_objects":8340,"degraded_total":112,"degraded_ratio":74.464285714285708,"recovering_objects_per_sec":81,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":973,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:03.395 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:03.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:03.688 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:03.688 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:03.705 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-04-26T02:38:03.125920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670004,"kb_used_data":1928,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817356,"statfs":{"total":386547056640,"available":383812972544,"internally_reserved":0,"allocated":1974272,"data_stored":851144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":973,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.313497"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.073495+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:37:47.073495+0000","last_peered":"2024-04-26T02:37:47.073495+0000","last_clean":"2024-04-26T02:37:47.073495+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:37:47.073495+0000","last_undegraded":"2024-04-26T02:37:47.073495+0000","last_fullsized":"2024-04-26T02:37:47.073495+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15431,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.067170+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:37:47.067170+0000","last_peered":"2024-04-26T02:37:47.067170+0000","last_clean":"2024-04-26T02:37:47.067170+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:37:47.067170+0000","last_undegraded":"2024-04-26T02:37:47.067170+0000","last_fullsized":"2024-04-26T02:37:47.067170+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:48.631104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15813,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066952+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:37:47.066952+0000","last_peered":"2024-04-26T02:37:47.066952+0000","last_clean":"2024-04-26T02:37:47.066952+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:37:47.066952+0000","last_undegraded":"2024-04-26T02:37:47.066952+0000","last_fullsized":"2024-04-26T02:37:47.066952+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15538,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.065897+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:37:47.065897+0000","last_peered":"2024-04-26T02:37:47.065897+0000","last_clean":"2024-04-26T02:37:47.065897+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:37:47.065897+0000","last_undegraded":"2024-04-26T02:37:47.065897+0000","last_fullsized":"2024-04-26T02:37:47.065897+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15731,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.162747+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:37:47.162747+0000","last_peered":"2024-04-26T02:37:47.162747+0000","last_clean":"2024-04-26T02:37:47.162747+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:37:47.162747+0000","last_undegraded":"2024-04-26T02:37:47.162747+0000","last_fullsized":"2024-04-26T02:37:47.162747+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11891,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070757+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:37:47.070757+0000","last_peered":"2024-04-26T02:37:47.070757+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:37:47.070757+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.068785+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:37:47.068785+0000","last_peered":"2024-04-26T02:37:47.068785+0000","last_clean":"2024-04-26T02:37:47.068785+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:37:47.068785+0000","last_undegraded":"2024-04-26T02:37:47.068785+0000","last_fullsized":"2024-04-26T02:37:47.068785+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.625526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12277,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066611+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:37:47.066611+0000","last_peered":"2024-04-26T02:37:47.066611+0000","last_clean":"2024-04-26T02:37:47.066611+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:37:47.066611+0000","last_undegraded":"2024-04-26T02:37:47.066611+0000","last_fullsized":"2024-04-26T02:37:47.066611+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15861,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.070579+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:37:47.070579+0000","last_peered":"2024-04-26T02:37:47.070579+0000","last_clean":"2024-04-26T02:37:47.070579+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:37:47.070579+0000","last_undegraded":"2024-04-26T02:37:47.070579+0000","last_fullsized":"2024-04-26T02:37:47.070579+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:49.811008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":14619,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:58.497603+0000","last_change":"2024-04-26T02:37:45.081694+0000","last_active":"2024-04-26T02:37:58.497603+0000","last_peered":"2024-04-26T02:37:58.497603+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:37:58.497603+0000","last_undegraded":"2024-04-26T02:24:59.277203+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-571}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11985,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073432+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:37:47.073432+0000","last_peered":"2024-04-26T02:37:47.073432+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:37:47.073432+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11860,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.073673+0000","last_change":"2024-04-26T02:24:59.612337+0000","last_active":"2024-04-26T02:37:47.073673+0000","last_peered":"2024-04-26T02:37:47.073673+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:37:47.073673+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12019,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066074+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:37:47.066074+0000","last_peered":"2024-04-26T02:37:47.066074+0000","last_clean":"2024-04-26T02:37:47.066074+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:37:47.066074+0000","last_undegraded":"2024-04-26T02:37:47.066074+0000","last_fullsized":"2024-04-26T02:37:47.066074+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12402,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:37:47.070095+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:37:47.070095+0000","last_peered":"2024-04-26T02:37:47.070095+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:37:47.070095+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16098,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066438+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:37:47.066438+0000","last_peered":"2024-04-26T02:37:47.066438+0000","last_clean":"2024-04-26T02:37:47.066438+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:37:47.066438+0000","last_undegraded":"2024-04-26T02:37:47.066438+0000","last_fullsized":"2024-04-26T02:37:47.066438+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12052,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-26T02:37:47.066737+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:37:47.066737+0000","last_peered":"2024-04-26T02:37:47.066737+0000","last_clean":"2024-04-26T02:37:47.066737+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:37:47.066737+0000","last_undegraded":"2024-04-26T02:37:47.066737+0000","last_fullsized":"2024-04-26T02:37:47.066737+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:50.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686024,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685816,"statfs":{"total":96636764160,"available":95934275584,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019037,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600028,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771812,"statfs":{"total":96636764160,"available":96022335488,"internally_reserved":0,"allocated":610304,"data_stored":329522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215279,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694280,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677560,"statfs":{"total":96636764160,"available":95925821440,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:06.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:07.031 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:07.031 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:07.050 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-26T02:38:06.817727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661732,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825628,"statfs":{"total":386547056640,"available":383821443072,"internally_reserved":0,"allocated":2011136,"data_stored":863086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003001"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15815,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T02:38:03.129564+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:03.129564+0000","last_peered":"2024-04-26T02:38:03.129564+0000","last_clean":"2024-04-26T02:38:03.129564+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:03.129564+0000","last_undegraded":"2024-04-26T02:38:03.129564+0000","last_fullsized":"2024-04-26T02:38:03.129564+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15767,"reported_epoch":54,"state":"peering","last_fresh":"2024-04-26T02:38:02.578456+0000","last_change":"2024-04-26T02:38:03.134599+0000","last_active":"2024-04-26T02:38:03.134599+0000","last_peered":"2024-04-26T02:38:02.578456+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:38:02.578456+0000","last_undegraded":"2024-04-26T02:38:02.578456+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":12241,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:05.062870+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:05.062870+0000","last_peered":"2024-04-26T02:38:05.062870+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:05.062870+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591716,"kb_used_data":600,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780124,"statfs":{"total":96636764160,"available":96030846976,"internally_reserved":0,"allocated":614400,"data_stored":328642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:07.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:07.368 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:07.369 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:07.388 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-26T02:38:06.817727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661732,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825628,"statfs":{"total":386547056640,"available":383821443072,"internally_reserved":0,"allocated":2011136,"data_stored":863086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003001"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15815,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T02:38:03.129564+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:03.129564+0000","last_peered":"2024-04-26T02:38:03.129564+0000","last_clean":"2024-04-26T02:38:03.129564+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:03.129564+0000","last_undegraded":"2024-04-26T02:38:03.129564+0000","last_fullsized":"2024-04-26T02:38:03.129564+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15767,"reported_epoch":54,"state":"peering","last_fresh":"2024-04-26T02:38:02.578456+0000","last_change":"2024-04-26T02:38:03.134599+0000","last_active":"2024-04-26T02:38:03.134599+0000","last_peered":"2024-04-26T02:38:02.578456+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:38:02.578456+0000","last_undegraded":"2024-04-26T02:38:02.578456+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":12241,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:05.062870+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:05.062870+0000","last_peered":"2024-04-26T02:38:05.062870+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:05.062870+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591716,"kb_used_data":600,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780124,"statfs":{"total":96636764160,"available":96030846976,"internally_reserved":0,"allocated":614400,"data_stored":328642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:07.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:07.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11895, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135432+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:04.135432+0000', 'last_peered': '2024-04-26T02:38:04.135432+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:04.135432+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:07.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-04-26T02:38:07.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "40'11562", 'reported_seq': 15767, 'reported_epoch': 54, 'state': 'peering', 'last_fresh': '2024-04-26T02:38:02.578456+0000', 'last_change': '2024-04-26T02:38:03.134599+0000', 'last_active': '2024-04-26T02:38:03.134599+0000', 'last_peered': '2024-04-26T02:38:02.578456+0000', 'last_clean': '2024-04-26T02:23:43.618831+0000', 'last_became_active': '2024-04-26T02:24:59.616504+0000', 'last_became_peered': '2024-04-26T02:24:59.616504+0000', 'last_unstale': '2024-04-26T02:38:02.578456+0000', 'last_undegraded': '2024-04-26T02:38:02.578456+0000', 'last_fullsized': '2024-04-26T02:24:59.230302+0000', 'mapping_epoch': 55, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11560, 'num_write_kb': 46246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2041, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11989, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135200+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:38:04.135200+0000', 'last_peered': '2024-04-26T02:38:04.135200+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:04.135200+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 12241, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:05.062870+0000', 'last_change': '2024-04-26T02:38:02.579617+0000', 'last_active': '2024-04-26T02:38:05.062870+0000', 'last_peered': '2024-04-26T02:38:05.062870+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:38:05.062870+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1711, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 279, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:07.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12406, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.134718+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:04.134718+0000', 'last_peered': '2024-04-26T02:38:04.134718+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:04.134718+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:07.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:07.744 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:07.744 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7579/112 objects degraded (6766.964%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":884,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725613568,"bytes_avail":383821443072,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":7579,"degraded_total":112,"degraded_ratio":67.669642857142861,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:07.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:08.085 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:08.086 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:08.105 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-04-26T02:38:06.817727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661732,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825628,"statfs":{"total":386547056640,"available":383821443072,"internally_reserved":0,"allocated":2011136,"data_stored":863086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003001"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15815,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T02:38:03.129564+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:03.129564+0000","last_peered":"2024-04-26T02:38:03.129564+0000","last_clean":"2024-04-26T02:38:03.129564+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:03.129564+0000","last_undegraded":"2024-04-26T02:38:03.129564+0000","last_fullsized":"2024-04-26T02:38:03.129564+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15767,"reported_epoch":54,"state":"peering","last_fresh":"2024-04-26T02:38:02.578456+0000","last_change":"2024-04-26T02:38:03.134599+0000","last_active":"2024-04-26T02:38:03.134599+0000","last_peered":"2024-04-26T02:38:02.578456+0000","last_clean":"2024-04-26T02:23:43.618831+0000","last_became_active":"2024-04-26T02:24:59.616504+0000","last_became_peered":"2024-04-26T02:24:59.616504+0000","last_unstale":"2024-04-26T02:38:02.578456+0000","last_undegraded":"2024-04-26T02:38:02.578456+0000","last_fullsized":"2024-04-26T02:24:59.230302+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":12241,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:05.062870+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:05.062870+0000","last_peered":"2024-04-26T02:38:05.062870+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:05.062870+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1710}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23144,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689672,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682168,"statfs":{"total":96636764160,"available":95930540032,"internally_reserved":0,"allocated":454656,"data_stored":173874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019038,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591716,"kb_used_data":600,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780124,"statfs":{"total":96636764160,"available":96030846976,"internally_reserved":0,"allocated":614400,"data_stored":328642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:11.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:11.447 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:11.448 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:11.466 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-26T02:38:10.819068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003000"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13043,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:10.065723+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:10.065723+0000","last_peered":"2024-04-26T02:38:10.065723+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:10.065723+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:11.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:11.758 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:11.758 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:11.776 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-26T02:38:10.819068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003000"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13043,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:10.065723+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:10.065723+0000","last_peered":"2024-04-26T02:38:10.065723+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:10.065723+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11895, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135432+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:04.135432+0000', 'last_peered': '2024-04-26T02:38:04.135432+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:04.135432+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11989, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135200+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:38:04.135200+0000', 'last_peered': '2024-04-26T02:38:04.135200+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:04.135200+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 13043, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:10.065723+0000', 'last_change': '2024-04-26T02:38:02.579617+0000', 'last_active': '2024-04-26T02:38:10.065723+0000', 'last_peered': '2024-04-26T02:38:10.065723+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:38:10.065723+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 680, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:11.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:11.778 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12406, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.134718+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:04.134718+0000', 'last_peered': '2024-04-26T02:38:04.134718+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:04.134718+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:11.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:12.134 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:12.134 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7178/112 objects degraded (6408.929%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":888,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2729897984,"bytes_avail":383817158656,"bytes_total":386547056640,"degraded_objects":7178,"degraded_total":112,"degraded_ratio":64.089285714285708,"recovering_objects_per_sec":145,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:12.153 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:12.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:12.502 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:12.502 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:12.520 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-04-26T02:38:10.819068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003000"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15435,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139028+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139028+0000","last_peered":"2024-04-26T02:38:04.139028+0000","last_clean":"2024-04-26T02:38:04.139028+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139028+0000","last_undegraded":"2024-04-26T02:38:04.139028+0000","last_fullsized":"2024-04-26T02:38:04.139028+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:04.636549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13043,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:10.065723+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:10.065723+0000","last_peered":"2024-04-26T02:38:10.065723+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:10.065723+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:15.522 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:15.841 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:15.864 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-04-26T02:38:14.820450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.692197"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13043,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:10.065723+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:10.065723+0000","last_peered":"2024-04-26T02:38:10.065723+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:10.065723+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215281,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:15.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:16.172 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:16.172 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:16.189 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-04-26T02:38:14.820450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.692197"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13043,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:10.065723+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:10.065723+0000","last_peered":"2024-04-26T02:38:10.065723+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:10.065723+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23545,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215281,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11895, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135432+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:04.135432+0000', 'last_peered': '2024-04-26T02:38:04.135432+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:04.135432+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11989, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135200+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:38:04.135200+0000', 'last_peered': '2024-04-26T02:38:04.135200+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:04.135200+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 13043, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:10.065723+0000', 'last_change': '2024-04-26T02:38:02.579617+0000', 'last_active': '2024-04-26T02:38:10.065723+0000', 'last_peered': '2024-04-26T02:38:10.065723+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:38:10.065723+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 680, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12406, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.134718+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:04.134718+0000', 'last_peered': '2024-04-26T02:38:04.134718+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:04.134718+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:16.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:16.552 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:16.553 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7178/112 objects degraded (6408.929%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2729897984,"bytes_avail":383817158656,"bytes_total":386547056640,"degraded_objects":7178,"degraded_total":112,"degraded_ratio":64.089285714285708,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":589,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:16.572 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:16.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:16.879 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:16.898 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-04-26T02:38:16.821171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004221"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13937,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:14.926500+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:14.926500+0000","last_peered":"2024-04-26T02:38:14.926500+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:14.926500+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:19.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:20.223 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:20.223 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:20.244 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-04-26T02:38:18.821936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004209"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13937,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:14.926500+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:14.926500+0000","last_peered":"2024-04-26T02:38:14.926500+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:14.926500+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:20.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:20.544 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:20.562 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-04-26T02:38:18.821936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665916,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821444,"statfs":{"total":386547056640,"available":383817158656,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-848,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004209"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":13937,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:14.926500+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:14.926500+0000","last_peered":"2024-04-26T02:38:14.926500+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:14.926500+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23992,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595880,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775960,"statfs":{"total":96636764160,"available":96026583040,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11895, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135432+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:04.135432+0000', 'last_peered': '2024-04-26T02:38:04.135432+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:04.135432+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11989, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135200+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:38:04.135200+0000', 'last_peered': '2024-04-26T02:38:04.135200+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:04.135200+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "40'11596", 'reported_seq': 13937, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:14.926500+0000', 'last_change': '2024-04-26T02:38:02.579617+0000', 'last_active': '2024-04-26T02:38:14.926500+0000', 'last_peered': '2024-04-26T02:38:14.926500+0000', 'last_clean': '2024-04-26T02:23:43.663851+0000', 'last_became_active': '2024-04-26T02:24:59.606468+0000', 'last_became_peered': '2024-04-26T02:24:59.606468+0000', 'last_unstale': '2024-04-26T02:38:14.926500+0000', 'last_undegraded': '2024-04-26T02:24:59.310607+0000', 'last_fullsized': '2024-04-26T02:24:59.230424+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6406, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11595, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -862}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12406, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.134718+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:04.134718+0000', 'last_peered': '2024-04-26T02:38:04.134718+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:04.134718+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:20.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:20.933 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:20.934 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6731/112 objects degraded (6009.821%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":897,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2729897984,"bytes_avail":383817158656,"bytes_total":386547056640,"degraded_objects":6731,"degraded_total":112,"degraded_ratio":60.098214285714285,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":848,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:20.952 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:20.952 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:21.287 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:21.305 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-04-26T02:38:20.822513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670076,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817284,"statfs":{"total":386547056640,"available":383812898816,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1334,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004083"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":14909,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:20.070352+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:20.070352+0000","last_peered":"2024-04-26T02:38:20.070352+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:20.070352+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600040,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771800,"statfs":{"total":96636764160,"available":96022323200,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215282,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:24.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:24.640 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:24.641 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:24.659 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-04-26T02:38:23.234882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670076,"kb_used_data":1988,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817284,"statfs":{"total":386547056640,"available":383812898816,"internally_reserved":0,"allocated":2035712,"data_stored":872692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.414432"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15436,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.139162+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:04.139162+0000","last_peered":"2024-04-26T02:38:04.139162+0000","last_clean":"2024-04-26T02:38:04.139162+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:04.139162+0000","last_undegraded":"2024-04-26T02:38:04.139162+0000","last_fullsized":"2024-04-26T02:38:04.139162+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:16.328363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15817,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.142160+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:04.142160+0000","last_peered":"2024-04-26T02:38:04.142160+0000","last_clean":"2024-04-26T02:38:04.142160+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:04.142160+0000","last_undegraded":"2024-04-26T02:38:04.142160+0000","last_fullsized":"2024-04-26T02:38:04.142160+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:08.982188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15542,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134094+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:04.134094+0000","last_peered":"2024-04-26T02:38:04.134094+0000","last_clean":"2024-04-26T02:38:04.134094+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:04.134094+0000","last_undegraded":"2024-04-26T02:38:04.134094+0000","last_fullsized":"2024-04-26T02:38:04.134094+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133786+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:04.133786+0000","last_peered":"2024-04-26T02:38:04.133786+0000","last_clean":"2024-04-26T02:38:04.133786+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:04.133786+0000","last_undegraded":"2024-04-26T02:38:04.133786+0000","last_fullsized":"2024-04-26T02:38:04.133786+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15777,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.315720+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:04.315720+0000","last_peered":"2024-04-26T02:38:04.315720+0000","last_clean":"2024-04-26T02:38:04.315720+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:04.315720+0000","last_undegraded":"2024-04-26T02:38:04.315720+0000","last_fullsized":"2024-04-26T02:38:04.315720+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":14909,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:20.070352+0000","last_change":"2024-04-26T02:38:02.579617+0000","last_active":"2024-04-26T02:38:20.070352+0000","last_peered":"2024-04-26T02:38:20.070352+0000","last_clean":"2024-04-26T02:23:43.663851+0000","last_became_active":"2024-04-26T02:24:59.606468+0000","last_became_peered":"2024-04-26T02:24:59.606468+0000","last_unstale":"2024-04-26T02:38:20.070352+0000","last_undegraded":"2024-04-26T02:24:59.310607+0000","last_fullsized":"2024-04-26T02:24:59.230424+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12023,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.133792+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:04.133792+0000","last_peered":"2024-04-26T02:38:04.133792+0000","last_clean":"2024-04-26T02:38:04.133792+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:04.133792+0000","last_undegraded":"2024-04-26T02:38:04.133792+0000","last_fullsized":"2024-04-26T02:38:04.133792+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16102,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134075+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:04.134075+0000","last_peered":"2024-04-26T02:38:04.134075+0000","last_clean":"2024-04-26T02:38:04.134075+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:04.134075+0000","last_undegraded":"2024-04-26T02:38:04.134075+0000","last_fullsized":"2024-04-26T02:38:04.134075+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12056,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138919+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:04.138919+0000","last_peered":"2024-04-26T02:38:04.138919+0000","last_clean":"2024-04-26T02:38:04.138919+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:04.138919+0000","last_undegraded":"2024-04-26T02:38:04.138919+0000","last_fullsized":"2024-04-26T02:38:04.138919+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.561698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24478,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689692,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682148,"statfs":{"total":96636764160,"available":95930519552,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019041,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600040,"kb_used_data":608,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771800,"statfs":{"total":96636764160,"available":96022323200,"internally_reserved":0,"allocated":622592,"data_stored":331837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:24.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:24.996 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:24.997 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:25.014 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-04-26T02:38:24.823521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661772,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825588,"statfs":{"total":386547056640,"available":383821402112,"internally_reserved":0,"allocated":2048000,"data_stored":869920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002350"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689696,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682144,"statfs":{"total":96636764160,"available":95930515456,"internally_reserved":0,"allocated":479232,"data_stored":183464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591732,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780108,"statfs":{"total":96636764160,"available":96030830592,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11895, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135432+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:04.135432+0000', 'last_peered': '2024-04-26T02:38:04.135432+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:04.135432+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 11989, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.135200+0000', 'last_change': '2024-04-26T02:24:59.613413+0000', 'last_active': '2024-04-26T02:38:04.135200+0000', 'last_peered': '2024-04-26T02:38:04.135200+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:04.135200+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:25.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12406, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:04.134718+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:04.134718+0000', 'last_peered': '2024-04-26T02:38:04.134718+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:04.134718+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:25.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:25.421 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:25.421 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5868/112 objects degraded (5239.286%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":901,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725654528,"bytes_avail":383821402112,"bytes_total":386547056640,"degraded_objects":5868,"degraded_total":112,"degraded_ratio":52.392857142857146},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:25.440 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:38:25.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:25.733 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:25.751 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-04-26T02:38:24.823521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661772,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825588,"statfs":{"total":386547056640,"available":383821402112,"internally_reserved":0,"allocated":2048000,"data_stored":869920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1036,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002350"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.138420+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:04.138420+0000","last_peered":"2024-04-26T02:38:04.138420+0000","last_clean":"2024-04-26T02:38:04.138420+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:04.138420+0000","last_undegraded":"2024-04-26T02:38:04.138420+0000","last_fullsized":"2024-04-26T02:38:04.138420+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15735,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.134523+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:04.134523+0000","last_peered":"2024-04-26T02:38:04.134523+0000","last_clean":"2024-04-26T02:38:04.134523+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:04.134523+0000","last_undegraded":"2024-04-26T02:38:04.134523+0000","last_fullsized":"2024-04-26T02:38:04.134523+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11895,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135432+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:04.135432+0000","last_peered":"2024-04-26T02:38:04.135432+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:04.135432+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16047,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.140797+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:04.140797+0000","last_peered":"2024-04-26T02:38:04.140797+0000","last_clean":"2024-04-26T02:38:04.140797+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:04.140797+0000","last_undegraded":"2024-04-26T02:38:04.140797+0000","last_fullsized":"2024-04-26T02:38:04.140797+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:07.502758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15865,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T02:38:04.135108+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:04.135108+0000","last_peered":"2024-04-26T02:38:04.135108+0000","last_clean":"2024-04-26T02:38:04.135108+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:04.135108+0000","last_undegraded":"2024-04-26T02:38:04.135108+0000","last_fullsized":"2024-04-26T02:38:04.135108+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.939086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":11989,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.135200+0000","last_change":"2024-04-26T02:24:59.613413+0000","last_active":"2024-04-26T02:38:04.135200+0000","last_peered":"2024-04-26T02:38:04.135200+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:04.135200+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12406,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:04.134718+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:04.134718+0000","last_peered":"2024-04-26T02:38:04.134718+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:04.134718+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22956,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686044,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685796,"statfs":{"total":96636764160,"available":95934255104,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689696,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682144,"statfs":{"total":96636764160,"available":95930515456,"internally_reserved":0,"allocated":479232,"data_stored":183464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591732,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780108,"statfs":{"total":96636764160,"available":96030830592,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694300,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677540,"statfs":{"total":96636764160,"available":95925800960,"internally_reserved":0,"allocated":471040,"data_stored":180285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:28.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:29.113 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:29.114 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:29.132 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-26T02:38:28.824670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661804,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825556,"statfs":{"total":386547056640,"available":383821369344,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002157"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15739,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243331+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243331+0000","last_peered":"2024-04-26T02:38:23.243331+0000","last_clean":"2024-04-26T02:38:23.243331+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243331+0000","last_undegraded":"2024-04-26T02:38:23.243331+0000","last_fullsized":"2024-04-26T02:38:23.243331+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":12440,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:25.065483+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:25.065483+0000","last_peered":"2024-04-26T02:38:25.065483+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:25.065483+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591732,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780108,"statfs":{"total":96636764160,"available":96030830592,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:29.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:29.443 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:29.444 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:29.462 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-26T02:38:28.824670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661804,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825556,"statfs":{"total":386547056640,"available":383821369344,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002157"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15739,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243331+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243331+0000","last_peered":"2024-04-26T02:38:23.243331+0000","last_clean":"2024-04-26T02:38:23.243331+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243331+0000","last_undegraded":"2024-04-26T02:38:23.243331+0000","last_fullsized":"2024-04-26T02:38:23.243331+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":12440,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:25.065483+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:25.065483+0000","last_peered":"2024-04-26T02:38:25.065483+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:25.065483+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591732,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780108,"statfs":{"total":96636764160,"available":96030830592,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11899, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.246245+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:23.246245+0000', 'last_peered': '2024-04-26T02:38:23.246245+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:23.246245+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 12440, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:25.065483+0000', 'last_change': '2024-04-26T02:38:21.996092+0000', 'last_active': '2024-04-26T02:38:25.065483+0000', 'last_peered': '2024-04-26T02:38:25.065483+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:25.065483+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1714, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 309, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:29.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12410, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.243687+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:23.243687+0000', 'last_peered': '2024-04-26T02:38:23.243687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:23.243687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:29.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:29.839 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:29.840 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5645/112 objects degraded (5040.179%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":906,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725687296,"bytes_avail":383821369344,"bytes_total":386547056640,"degraded_objects":5645,"degraded_total":112,"degraded_ratio":50.401785714285715},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:29.861 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:38:29.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:30.165 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:30.166 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:30.183 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-04-26T02:38:28.824670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661804,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825556,"statfs":{"total":386547056640,"available":383821369344,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002157"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15739,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243331+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243331+0000","last_peered":"2024-04-26T02:38:23.243331+0000","last_clean":"2024-04-26T02:38:23.243331+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243331+0000","last_undegraded":"2024-04-26T02:38:23.243331+0000","last_fullsized":"2024-04-26T02:38:23.243331+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":12440,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:25.065483+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:25.065483+0000","last_peered":"2024-04-26T02:38:25.065483+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:25.065483+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23179,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019042,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591732,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780108,"statfs":{"total":96636764160,"available":96030830592,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215284,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:33.185 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:33.522 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:33.539 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-04-26T02:38:32.825853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661796,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825564,"statfs":{"total":386547056640,"available":383821377536,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-864,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002895"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":13310,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:30.072689+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:30.072689+0000","last_peered":"2024-04-26T02:38:30.072689+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:30.072689+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591724,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780116,"statfs":{"total":96636764160,"available":96030838784,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215285,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:33.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:33.836 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:33.854 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-04-26T02:38:32.825853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661796,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825564,"statfs":{"total":386547056640,"available":383821377536,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-864,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002895"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":13310,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:30.072689+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:30.072689+0000","last_peered":"2024-04-26T02:38:30.072689+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:30.072689+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591724,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780116,"statfs":{"total":96636764160,"available":96030838784,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215285,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:33.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:33.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11899, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.246245+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:23.246245+0000', 'last_peered': '2024-04-26T02:38:23.246245+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:23.246245+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:33.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:33.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 13310, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:30.072689+0000', 'last_change': '2024-04-26T02:38:21.996092+0000', 'last_active': '2024-04-26T02:38:30.072689+0000', 'last_peered': '2024-04-26T02:38:30.072689+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:30.072689+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 744, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:33.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:33.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12410, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.243687+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:23.243687+0000', 'last_peered': '2024-04-26T02:38:23.243687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:23.243687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:33.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:34.243 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5210/112 objects degraded (4651.786%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":910,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725679104,"bytes_avail":383821377536,"bytes_total":386547056640,"degraded_objects":5210,"degraded_total":112,"degraded_ratio":46.517857142857146},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T02:37:58.814771+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:34.262 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:38:34.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:34.567 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:34.567 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:34.586 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-04-26T02:38:32.825853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661796,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825564,"statfs":{"total":386547056640,"available":383821377536,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-864,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002895"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":13310,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:30.072689+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:30.072689+0000","last_peered":"2024-04-26T02:38:30.072689+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:30.072689+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23614,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019043,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591724,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780116,"statfs":{"total":96636764160,"available":96030838784,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215285,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:37.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:37.922 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:37.922 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:37.941 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-04-26T02:38:36.827046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665956,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821404,"statfs":{"total":386547056640,"available":383817117696,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003524"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":14276,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:35.066266+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:35.066266+0000","last_peered":"2024-04-26T02:38:35.066266+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:35.066266+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595884,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775956,"statfs":{"total":96636764160,"available":96026578944,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:37.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:38.256 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:38.257 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:38.278 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-04-26T02:38:36.827046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665956,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821404,"statfs":{"total":386547056640,"available":383817117696,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003524"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":14276,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:35.066266+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:35.066266+0000","last_peered":"2024-04-26T02:38:35.066266+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:35.066266+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595884,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775956,"statfs":{"total":96636764160,"available":96026578944,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11899, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.246245+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:23.246245+0000', 'last_peered': '2024-04-26T02:38:23.246245+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:23.246245+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 14276, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:35.066266+0000', 'last_change': '2024-04-26T02:38:21.996092+0000', 'last_active': '2024-04-26T02:38:35.066266+0000', 'last_peered': '2024-04-26T02:38:35.066266+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:35.066266+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 796, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1227, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -793}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12410, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.243687+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:23.243687+0000', 'last_peered': '2024-04-26T02:38:23.243687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:23.243687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:38.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:38.643 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:38.644 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4727/112 objects degraded (4220.536%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":915,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2729938944,"bytes_avail":383817117696,"bytes_total":386547056640,"degraded_objects":4727,"degraded_total":112,"degraded_ratio":42.205357142857146,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:38.663 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:38.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:38.958 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:38.976 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-04-26T02:38:38.827821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2665956,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821404,"statfs":{"total":386547056640,"available":383817117696,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":918,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12285,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235053+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235053+0000","last_peered":"2024-04-26T02:38:23.235053+0000","last_clean":"2024-04-26T02:38:23.235053+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235053+0000","last_undegraded":"2024-04-26T02:38:23.235053+0000","last_fullsized":"2024-04-26T02:38:23.235053+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":14276,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:35.066266+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:35.066266+0000","last_peered":"2024-04-26T02:38:35.066266+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:35.066266+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24097,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595884,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775956,"statfs":{"total":96636764160,"available":96026578944,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215286,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:41.978 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:42.321 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:42.341 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-04-26T02:38:41.920361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670116,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817244,"statfs":{"total":386547056640,"available":383812857856,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.095158"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12286,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235190+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235190+0000","last_peered":"2024-04-26T02:38:23.235190+0000","last_clean":"2024-04-26T02:38:23.235190+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235190+0000","last_undegraded":"2024-04-26T02:38:23.235190+0000","last_fullsized":"2024-04-26T02:38:23.235190+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:41.408376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15452,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:40.076484+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:40.076484+0000","last_peered":"2024-04-26T02:38:40.076484+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:40.076484+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600044,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771796,"statfs":{"total":96636764160,"available":96022319104,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:42.342 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:42.668 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:42.688 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-04-26T02:38:41.920361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670116,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817244,"statfs":{"total":386547056640,"available":383812857856,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.095158"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12286,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235190+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235190+0000","last_peered":"2024-04-26T02:38:23.235190+0000","last_clean":"2024-04-26T02:38:23.235190+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235190+0000","last_undegraded":"2024-04-26T02:38:23.235190+0000","last_fullsized":"2024-04-26T02:38:23.235190+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:41.408376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15452,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:40.076484+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:40.076484+0000","last_peered":"2024-04-26T02:38:40.076484+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:40.076484+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600044,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771796,"statfs":{"total":96636764160,"available":96022319104,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:42.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:42.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 11899, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.246245+0000', 'last_change': '2024-04-26T02:24:59.615715+0000', 'last_active': '2024-04-26T02:38:23.246245+0000', 'last_peered': '2024-04-26T02:38:23.246245+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:23.246245+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1950, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 101, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:42.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-04-26T02:38:42.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "40'11716", 'reported_seq': 15452, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:40.076484+0000', 'last_change': '2024-04-26T02:38:21.996092+0000', 'last_active': '2024-04-26T02:38:40.076484+0000', 'last_peered': '2024-04-26T02:38:40.076484+0000', 'last_clean': '2024-04-26T02:23:51.598727+0000', 'last_became_active': '2024-04-26T02:24:59.605573+0000', 'last_became_peered': '2024-04-26T02:24:59.605573+0000', 'last_unstale': '2024-04-26T02:38:40.076484+0000', 'last_undegraded': '2024-04-26T02:24:59.313965+0000', 'last_fullsized': '2024-04-26T02:24:59.228279+0000', 'mapping_epoch': 31, 'log_start': "28'1700", 'ondisk_log_start': "28'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19218, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11713, 'num_write_kb': 46860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1815, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -205}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:42.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:42.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12410, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:23.243687+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:23.243687+0000', 'last_peered': '2024-04-26T02:38:23.243687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:23.243687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:42.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:43.109 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4139/112 objects degraded (3695.536%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":919,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734198784,"bytes_avail":383812857856,"bytes_total":386547056640,"degraded_objects":4139,"degraded_total":112,"degraded_ratio":36.955357142857146,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1071,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:43.132 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:43.132 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:43.447 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:43.448 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:43.466 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-04-26T02:38:42.930117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670116,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817244,"statfs":{"total":386547056640,"available":383812857856,"internally_reserved":0,"allocated":2088960,"data_stored":885752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.103766"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15440,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235428+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:23.235428+0000","last_peered":"2024-04-26T02:38:23.235428+0000","last_clean":"2024-04-26T02:38:23.235428+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:23.235428+0000","last_undegraded":"2024-04-26T02:38:23.235428+0000","last_fullsized":"2024-04-26T02:38:23.235428+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:27.416867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15821,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244169+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:23.244169+0000","last_peered":"2024-04-26T02:38:23.244169+0000","last_clean":"2024-04-26T02:38:23.244169+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:23.244169+0000","last_undegraded":"2024-04-26T02:38:23.244169+0000","last_fullsized":"2024-04-26T02:38:23.244169+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.086324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15546,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233344+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:23.233344+0000","last_peered":"2024-04-26T02:38:23.233344+0000","last_clean":"2024-04-26T02:38:23.233344+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:23.233344+0000","last_undegraded":"2024-04-26T02:38:23.233344+0000","last_fullsized":"2024-04-26T02:38:23.233344+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12286,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235190+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:23.235190+0000","last_peered":"2024-04-26T02:38:23.235190+0000","last_clean":"2024-04-26T02:38:23.235190+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:23.235190+0000","last_undegraded":"2024-04-26T02:38:23.235190+0000","last_fullsized":"2024-04-26T02:38:23.235190+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:41.408376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15782,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.244585+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:23.244585+0000","last_peered":"2024-04-26T02:38:23.244585+0000","last_clean":"2024-04-26T02:38:23.244585+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:23.244585+0000","last_undegraded":"2024-04-26T02:38:23.244585+0000","last_fullsized":"2024-04-26T02:38:23.244585+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:26.135343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15452,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:40.076484+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:40.076484+0000","last_peered":"2024-04-26T02:38:40.076484+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:40.076484+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11845,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.401403+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:23.401403+0000","last_peered":"2024-04-26T02:38:23.401403+0000","last_clean":"2024-04-26T02:38:23.401403+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:23.401403+0000","last_undegraded":"2024-04-26T02:38:23.401403+0000","last_fullsized":"2024-04-26T02:38:23.401403+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12027,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.233747+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:23.233747+0000","last_peered":"2024-04-26T02:38:23.233747+0000","last_clean":"2024-04-26T02:38:23.233747+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:23.233747+0000","last_undegraded":"2024-04-26T02:38:23.233747+0000","last_fullsized":"2024-04-26T02:38:23.233747+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.234648+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:23.234648+0000","last_peered":"2024-04-26T02:38:23.234648+0000","last_clean":"2024-04-26T02:38:23.234648+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:23.234648+0000","last_undegraded":"2024-04-26T02:38:23.234648+0000","last_fullsized":"2024-04-26T02:38:23.234648+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12060,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.235344+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:23.235344+0000","last_peered":"2024-04-26T02:38:23.235344+0000","last_clean":"2024-04-26T02:38:23.235344+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:23.235344+0000","last_undegraded":"2024-04-26T02:38:23.235344+0000","last_fullsized":"2024-04-26T02:38:23.235344+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:25.354472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600044,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771796,"statfs":{"total":96636764160,"available":96022319104,"internally_reserved":0,"allocated":626688,"data_stored":325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:46.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:46.813 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:46.814 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:46.832 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-04-26T02:38:44.829404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661832,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825528,"statfs":{"total":386547056640,"available":383821340672,"internally_reserved":0,"allocated":2121728,"data_stored":898278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002358"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15662,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.246130+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:23.246130+0000","last_peered":"2024-04-26T02:38:23.246130+0000","last_clean":"2024-04-26T02:38:23.246130+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:23.246130+0000","last_undegraded":"2024-04-26T02:38:23.246130+0000","last_fullsized":"2024-04-26T02:38:23.246130+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15550,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931082+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931082+0000","last_peered":"2024-04-26T02:38:42.931082+0000","last_clean":"2024-04-26T02:38:42.931082+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931082+0000","last_undegraded":"2024-04-26T02:38:42.931082+0000","last_fullsized":"2024-04-26T02:38:42.931082+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15740,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243487+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:23.243487+0000","last_peered":"2024-04-26T02:38:23.243487+0000","last_clean":"2024-04-26T02:38:23.243487+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:23.243487+0000","last_undegraded":"2024-04-26T02:38:23.243487+0000","last_fullsized":"2024-04-26T02:38:23.243487+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:33.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":11899,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.246245+0000","last_change":"2024-04-26T02:24:59.615715+0000","last_active":"2024-04-26T02:38:23.246245+0000","last_peered":"2024-04-26T02:38:23.246245+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:23.246245+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16051,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.241828+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:23.241828+0000","last_peered":"2024-04-26T02:38:23.241828+0000","last_clean":"2024-04-26T02:38:23.241828+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:23.241828+0000","last_undegraded":"2024-04-26T02:38:23.241828+0000","last_fullsized":"2024-04-26T02:38:23.241828+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.415940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-26T02:38:23.243673+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:23.243673+0000","last_peered":"2024-04-26T02:38:23.243673+0000","last_clean":"2024-04-26T02:38:23.243673+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:23.243673+0000","last_undegraded":"2024-04-26T02:38:23.243673+0000","last_fullsized":"2024-04-26T02:38:23.243673+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:23.723808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15452,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:40.076484+0000","last_change":"2024-04-26T02:38:21.996092+0000","last_active":"2024-04-26T02:38:40.076484+0000","last_peered":"2024-04-26T02:38:40.076484+0000","last_clean":"2024-04-26T02:23:51.598727+0000","last_became_active":"2024-04-26T02:24:59.605573+0000","last_became_peered":"2024-04-26T02:24:59.605573+0000","last_unstale":"2024-04-26T02:38:40.076484+0000","last_undegraded":"2024-04-26T02:24:59.313965+0000","last_fullsized":"2024-04-26T02:24:59.228279+0000","mapping_epoch":31,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12410,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:23.243687+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:23.243687+0000","last_peered":"2024-04-26T02:38:23.243687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:23.243687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24685,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686056,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685784,"statfs":{"total":96636764160,"available":95934242816,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591740,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780100,"statfs":{"total":96636764160,"available":96030822400,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694312,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677528,"statfs":{"total":96636764160,"available":95925788672,"internally_reserved":0,"allocated":487424,"data_stored":186622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:46.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:47.151 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:47.152 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:47.170 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-04-26T02:38:46.830033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25355,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661872,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825488,"statfs":{"total":386547056640,"available":383821299712,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002211"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15550,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931082+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931082+0000","last_peered":"2024-04-26T02:38:42.931082+0000","last_clean":"2024-04-26T02:38:42.931082+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931082+0000","last_undegraded":"2024-04-26T02:38:42.931082+0000","last_fullsized":"2024-04-26T02:38:42.931082+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":12828,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:45.079033+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:45.079033+0000","last_peered":"2024-04-26T02:38:45.079033+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:45.079033+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16055,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933618+0000","last_peered":"2024-04-26T02:38:42.933618+0000","last_clean":"2024-04-26T02:38:42.933618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933618+0000","last_undegraded":"2024-04-26T02:38:42.933618+0000","last_fullsized":"2024-04-26T02:38:42.933618+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25355,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591740,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780100,"statfs":{"total":96636764160,"available":96030822400,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:47.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:47.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 12828, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:45.079033+0000', 'last_change': '2024-04-26T02:38:41.642832+0000', 'last_active': '2024-04-26T02:38:45.079033+0000', 'last_peered': '2024-04-26T02:38:45.079033+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:45.079033+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1488, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 563, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1488}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:47.171 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:47.171 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12414, 'reported_epoch': 60, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:42.934253+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:42.934253+0000', 'last_peered': '2024-04-26T02:38:42.934253+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:42.934253+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:47.171 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:47.550 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:47.551 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3469/112 objects degraded (3097.321%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":924,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725756928,"bytes_avail":383821299712,"bytes_total":386547056640,"degraded_objects":3469,"degraded_total":112,"degraded_ratio":30.973214285714285,"recovering_objects_per_sec":157,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:47.568 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:47.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:47.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:47.883 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:47.902 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-04-26T02:38:46.830033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25355,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661872,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825488,"statfs":{"total":386547056640,"available":383821299712,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002211"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15550,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931082+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931082+0000","last_peered":"2024-04-26T02:38:42.931082+0000","last_clean":"2024-04-26T02:38:42.931082+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931082+0000","last_undegraded":"2024-04-26T02:38:42.931082+0000","last_fullsized":"2024-04-26T02:38:42.931082+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":12828,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:45.079033+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:45.079033+0000","last_peered":"2024-04-26T02:38:45.079033+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:45.079033+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16055,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933618+0000","last_peered":"2024-04-26T02:38:42.933618+0000","last_clean":"2024-04-26T02:38:42.933618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933618+0000","last_undegraded":"2024-04-26T02:38:42.933618+0000","last_fullsized":"2024-04-26T02:38:42.933618+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25355,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019046,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591740,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780100,"statfs":{"total":96636764160,"available":96030822400,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:50.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:51.251 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:51.269 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-04-26T02:38:50.831208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666032,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821328,"statfs":{"total":386547056640,"available":383817039872,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.910847"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16055,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933618+0000","last_peered":"2024-04-26T02:38:42.933618+0000","last_clean":"2024-04-26T02:38:42.933618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933618+0000","last_undegraded":"2024-04-26T02:38:42.933618+0000","last_fullsized":"2024-04-26T02:38:42.933618+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595900,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775940,"statfs":{"total":96636764160,"available":96026562560,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:51.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:51.614 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:51.614 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:51.633 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-04-26T02:38:50.831208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666032,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821328,"statfs":{"total":386547056640,"available":383817039872,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.910847"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16055,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933618+0000","last_peered":"2024-04-26T02:38:42.933618+0000","last_clean":"2024-04-26T02:38:42.933618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933618+0000","last_undegraded":"2024-04-26T02:38:42.933618+0000","last_fullsized":"2024-04-26T02:38:42.933618+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595900,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775940,"statfs":{"total":96636764160,"available":96026562560,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:51.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:51.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 13646, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:50.076981+0000', 'last_change': '2024-04-26T02:38:41.642832+0000', 'last_active': '2024-04-26T02:38:50.076981+0000', 'last_peered': '2024-04-26T02:38:50.076981+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:50.076981+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 972, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1079}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:51.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:51.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12414, 'reported_epoch': 60, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:42.934253+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:42.934253+0000', 'last_peered': '2024-04-26T02:38:42.934253+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:42.934253+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:51.635 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:52.003 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:52.003 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3060/112 objects degraded (2732.143%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":928,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2730016768,"bytes_avail":383817039872,"bytes_total":386547056640,"degraded_objects":3060,"degraded_total":112,"degraded_ratio":27.321428571428573,"recovering_objects_per_sec":121,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:52.021 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:52.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:52.336 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:52.336 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:52.353 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-04-26T02:38:50.831208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666032,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821328,"statfs":{"total":386547056640,"available":383817039872,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.910847"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16055,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933618+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933618+0000","last_peered":"2024-04-26T02:38:42.933618+0000","last_clean":"2024-04-26T02:38:42.933618+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933618+0000","last_undegraded":"2024-04-26T02:38:42.933618+0000","last_fullsized":"2024-04-26T02:38:42.933618+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019047,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595900,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775940,"statfs":{"total":96636764160,"available":96026562560,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215288,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:55.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:55.675 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:55.675 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:55.693 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-04-26T02:38:54.832617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670192,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817168,"statfs":{"total":386547056640,"available":383812780032,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.902499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600060,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771780,"statfs":{"total":96636764160,"available":96022302720,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:55.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:55.992 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:55.992 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:56.010 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-04-26T02:38:54.832617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670192,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817168,"statfs":{"total":386547056640,"available":383812780032,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.902499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600060,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771780,"statfs":{"total":96636764160,"available":96022302720,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:56.011 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:38:56.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 13646, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:50.076981+0000', 'last_change': '2024-04-26T02:38:41.642832+0000', 'last_active': '2024-04-26T02:38:50.076981+0000', 'last_peered': '2024-04-26T02:38:50.076981+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:50.076981+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 972, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1079}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:56.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:38:56.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12414, 'reported_epoch': 60, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:38:42.934253+0000', 'last_change': '2024-04-26T02:24:59.648380+0000', 'last_active': '2024-04-26T02:38:42.934253+0000', 'last_peered': '2024-04-26T02:38:42.934253+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:38:42.934253+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1981, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:38:56.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:38:56.391 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:56.391 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3060/112 objects degraded (2732.143%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":932,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734276608,"bytes_avail":383812780032,"bytes_total":386547056640,"degraded_objects":3060,"degraded_total":112,"degraded_ratio":27.321428571428573,"recovering_objects_per_sec":90,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:38:56.411 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:38:56.411 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:38:56.729 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:38:56.729 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:38:56.748 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-04-26T02:38:54.832617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670192,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817168,"statfs":{"total":386547056640,"available":383812780032,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.902499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":13646,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:50.076981+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:50.076981+0000","last_peered":"2024-04-26T02:38:50.076981+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:50.076981+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25764,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600060,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771780,"statfs":{"total":96636764160,"available":96022302720,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215289,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:38:59.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:00.072 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:00.072 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:00.091 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-04-26T02:38:58.833762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26270,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670192,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817168,"statfs":{"total":386547056640,"available":383812780032,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":915,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003728"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":14658,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:55.086263+0000","last_change":"2024-04-26T02:38:41.642832+0000","last_active":"2024-04-26T02:38:55.086263+0000","last_peered":"2024-04-26T02:38:55.086263+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:55.086263+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15873,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934284+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934284+0000","last_peered":"2024-04-26T02:38:42.934284+0000","last_clean":"2024-04-26T02:38:42.934284+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934284+0000","last_undegraded":"2024-04-26T02:38:42.934284+0000","last_fullsized":"2024-04-26T02:38:42.934284+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.564084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12414,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-26T02:38:42.934253+0000","last_change":"2024-04-26T02:24:59.648380+0000","last_active":"2024-04-26T02:38:42.934253+0000","last_peered":"2024-04-26T02:38:42.934253+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:38:42.934253+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1977}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26270,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019048,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600060,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771780,"statfs":{"total":96636764160,"available":96022302720,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215290,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:00.092 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:00.458 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:00.459 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:00.480 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-04-26T02:39:00.203279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657712,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829648,"statfs":{"total":386547056640,"available":383825559552,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1543,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.372705"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15805,"reported_epoch":60,"state":"active+undersized+remapped","last_fresh":"2024-04-26T02:38:59.295028+0000","last_change":"2024-04-26T02:38:59.295670+0000","last_active":"2024-04-26T02:38:59.295028+0000","last_peered":"2024-04-26T02:38:59.295028+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:59.295028+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15874,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934335+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934335+0000","last_peered":"2024-04-26T02:38:42.934335+0000","last_clean":"2024-04-26T02:38:42.934335+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934335+0000","last_undegraded":"2024-04-26T02:38:42.934335+0000","last_fullsized":"2024-04-26T02:38:42.934335+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12525,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:00.084687+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:00.084687+0000","last_peered":"2024-04-26T02:39:00.084687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:00.084687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587580,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784260,"statfs":{"total":96636764160,"available":96035082240,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215290,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:00.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:39:00.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 15805, 'reported_epoch': 60, 'state': 'active+undersized+remapped', 'last_fresh': '2024-04-26T02:38:59.295028+0000', 'last_change': '2024-04-26T02:38:59.295670+0000', 'last_active': '2024-04-26T02:38:59.295028+0000', 'last_peered': '2024-04-26T02:38:59.295028+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:59.295028+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2051, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '1(3)', '2(2)'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:00.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:00.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12525, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:39:00.084687+0000', 'last_change': '2024-04-26T02:38:59.298002+0000', 'last_active': '2024-04-26T02:39:00.084687+0000', 'last_peered': '2024-04-26T02:39:00.084687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:39:00.084687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 149, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:00.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:00.882 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2554/112 objects degraded (2280.357%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":937,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734276608,"bytes_avail":383812780032,"bytes_total":386547056640,"degraded_objects":2554,"degraded_total":112,"degraded_ratio":22.803571428571427,"recovering_objects_per_sec":76,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":915,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:00.902 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:39:00.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:01.228 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:01.228 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:01.250 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-04-26T02:39:01.209371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657712,"kb_used_data":2104,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829648,"statfs":{"total":386547056640,"available":383825559552,"internally_reserved":0,"allocated":2154496,"data_stored":910804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.377475"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15805,"reported_epoch":60,"state":"active+undersized+remapped","last_fresh":"2024-04-26T02:38:59.295028+0000","last_change":"2024-04-26T02:38:59.295670+0000","last_active":"2024-04-26T02:38:59.295028+0000","last_peered":"2024-04-26T02:38:59.295028+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:59.295028+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16056,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.933675+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:38:42.933675+0000","last_peered":"2024-04-26T02:38:42.933675+0000","last_clean":"2024-04-26T02:38:42.933675+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:38:42.933675+0000","last_undegraded":"2024-04-26T02:38:42.933675+0000","last_fullsized":"2024-04-26T02:38:42.933675+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:54.632936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15874,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934335+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934335+0000","last_peered":"2024-04-26T02:38:42.934335+0000","last_clean":"2024-04-26T02:38:42.934335+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934335+0000","last_undegraded":"2024-04-26T02:38:42.934335+0000","last_fullsized":"2024-04-26T02:38:42.934335+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12525,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:00.084687+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:00.084687+0000","last_peered":"2024-04-26T02:39:00.084687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:00.084687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587580,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784260,"statfs":{"total":96636764160,"available":96035082240,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215290,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694332,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677508,"statfs":{"total":96636764160,"available":95925768192,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:04.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:04.567 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:04.567 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:04.585 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-04-26T02:39:02.834850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657728,"kb_used_data":2120,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829632,"statfs":{"total":386547056640,"available":383825543168,"internally_reserved":0,"allocated":2170880,"data_stored":916993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002232"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15444,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931397+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:38:42.931397+0000","last_peered":"2024-04-26T02:38:42.931397+0000","last_clean":"2024-04-26T02:38:42.931397+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:38:42.931397+0000","last_undegraded":"2024-04-26T02:38:42.931397+0000","last_fullsized":"2024-04-26T02:38:42.931397+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:43.386527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15825,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934387+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:38:42.934387+0000","last_peered":"2024-04-26T02:38:42.934387+0000","last_clean":"2024-04-26T02:38:42.934387+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:38:42.934387+0000","last_undegraded":"2024-04-26T02:38:42.934387+0000","last_fullsized":"2024-04-26T02:38:42.934387+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.147806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931150+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:38:42.931150+0000","last_peered":"2024-04-26T02:38:42.931150+0000","last_clean":"2024-04-26T02:38:42.931150+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:38:42.931150+0000","last_undegraded":"2024-04-26T02:38:42.931150+0000","last_fullsized":"2024-04-26T02:38:42.931150+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:52.375695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15805,"reported_epoch":60,"state":"active+undersized+remapped","last_fresh":"2024-04-26T02:38:59.295028+0000","last_change":"2024-04-26T02:38:59.295670+0000","last_active":"2024-04-26T02:38:59.295028+0000","last_peered":"2024-04-26T02:38:59.295028+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:59.295028+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12290,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931460+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:38:42.931460+0000","last_peered":"2024-04-26T02:38:42.931460+0000","last_clean":"2024-04-26T02:38:42.931460+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:38:42.931460+0000","last_undegraded":"2024-04-26T02:38:42.931460+0000","last_fullsized":"2024-04-26T02:38:42.931460+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15874,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934335+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934335+0000","last_peered":"2024-04-26T02:38:42.934335+0000","last_clean":"2024-04-26T02:38:42.934335+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934335+0000","last_undegraded":"2024-04-26T02:38:42.934335+0000","last_fullsized":"2024-04-26T02:38:42.934335+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15786,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934504+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:38:42.934504+0000","last_peered":"2024-04-26T02:38:42.934504+0000","last_clean":"2024-04-26T02:38:42.934504+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:38:42.934504+0000","last_undegraded":"2024-04-26T02:38:42.934504+0000","last_fullsized":"2024-04-26T02:38:42.934504+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:47.144422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931802+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:38:42.931802+0000","last_peered":"2024-04-26T02:38:42.931802+0000","last_clean":"2024-04-26T02:38:42.931802+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:38:42.931802+0000","last_undegraded":"2024-04-26T02:38:42.931802+0000","last_fullsized":"2024-04-26T02:38:42.931802+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:45.419864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12031,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931180+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:38:42.931180+0000","last_peered":"2024-04-26T02:38:42.931180+0000","last_clean":"2024-04-26T02:38:42.931180+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:38:42.931180+0000","last_undegraded":"2024-04-26T02:38:42.931180+0000","last_fullsized":"2024-04-26T02:38:42.931180+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12525,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:00.084687+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:00.084687+0000","last_peered":"2024-04-26T02:39:00.084687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:00.084687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16110,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931304+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:38:42.931304+0000","last_peered":"2024-04-26T02:38:42.931304+0000","last_clean":"2024-04-26T02:38:42.931304+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:38:42.931304+0000","last_undegraded":"2024-04-26T02:38:42.931304+0000","last_fullsized":"2024-04-26T02:38:42.931304+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12064,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.931555+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:38:42.931555+0000","last_peered":"2024-04-26T02:38:42.931555+0000","last_clean":"2024-04-26T02:38:42.931555+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:38:42.931555+0000","last_undegraded":"2024-04-26T02:38:42.931555+0000","last_fullsized":"2024-04-26T02:38:42.931555+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:46.417239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689724,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682116,"statfs":{"total":96636764160,"available":95930486784,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019049,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587580,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784260,"statfs":{"total":96636764160,"available":96035082240,"internally_reserved":0,"allocated":643072,"data_stored":332149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:04.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:04.901 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:04.901 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:04.921 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-04-26T02:39:04.835538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661912,"kb_used_data":2144,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825448,"statfs":{"total":386547056640,"available":383821258752,"internally_reserved":0,"allocated":2195456,"data_stored":921179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15805,"reported_epoch":60,"state":"active+undersized+remapped","last_fresh":"2024-04-26T02:38:59.295028+0000","last_change":"2024-04-26T02:38:59.295670+0000","last_active":"2024-04-26T02:38:59.295028+0000","last_peered":"2024-04-26T02:38:59.295028+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:59.295028+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15874,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934335+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934335+0000","last_peered":"2024-04-26T02:38:42.934335+0000","last_clean":"2024-04-26T02:38:42.934335+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934335+0000","last_undegraded":"2024-04-26T02:38:42.934335+0000","last_fullsized":"2024-04-26T02:38:42.934335+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12525,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:00.084687+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:00.084687+0000","last_peered":"2024-04-26T02:39:00.084687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:00.084687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591748,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780092,"statfs":{"total":96636764160,"available":96030814208,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:04.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-04-26T02:39:04.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "40'11600", 'reported_seq': 15805, 'reported_epoch': 60, 'state': 'active+undersized+remapped', 'last_fresh': '2024-04-26T02:38:59.295028+0000', 'last_change': '2024-04-26T02:38:59.295670+0000', 'last_active': '2024-04-26T02:38:59.295028+0000', 'last_peered': '2024-04-26T02:38:59.295028+0000', 'last_clean': '2024-04-26T02:23:40.559762+0000', 'last_became_active': '2024-04-26T02:24:59.607750+0000', 'last_became_peered': '2024-04-26T02:24:59.607750+0000', 'last_unstale': '2024-04-26T02:38:59.295028+0000', 'last_undegraded': '2024-04-26T02:24:59.296484+0000', 'last_fullsized': '2024-04-26T02:24:59.230081+0000', 'mapping_epoch': 31, 'log_start': "28'1500", 'ondisk_log_start': "28'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 46400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2051, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '1(3)', '2(2)'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:04.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:04.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12525, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:39:00.084687+0000', 'last_change': '2024-04-26T02:38:59.298002+0000', 'last_active': '2024-04-26T02:39:00.084687+0000', 'last_peered': '2024-04-26T02:39:00.084687+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:39:00.084687+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 149, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:04.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:05.295 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:05.295 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1926/112 objects degraded (1719.643%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":941,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725797888,"bytes_avail":383821258752,"bytes_total":386547056640,"degraded_objects":1926,"degraded_total":112,"degraded_ratio":17.196428571428573,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":628,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:05.314 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:39:05.314 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:05.621 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:05.621 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:05.639 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-04-26T02:39:04.835538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661912,"kb_used_data":2144,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825448,"statfs":{"total":386547056640,"available":383821258752,"internally_reserved":0,"allocated":2195456,"data_stored":921179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15666,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.935689+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:38:42.935689+0000","last_peered":"2024-04-26T02:38:42.935689+0000","last_clean":"2024-04-26T02:38:42.935689+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:38:42.935689+0000","last_undegraded":"2024-04-26T02:38:42.935689+0000","last_fullsized":"2024-04-26T02:38:42.935689+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15744,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934124+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:38:42.934124+0000","last_peered":"2024-04-26T02:38:42.934124+0000","last_clean":"2024-04-26T02:38:42.934124+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:38:42.934124+0000","last_undegraded":"2024-04-26T02:38:42.934124+0000","last_fullsized":"2024-04-26T02:38:42.934124+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:44.589344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15805,"reported_epoch":60,"state":"active+undersized+remapped","last_fresh":"2024-04-26T02:38:59.295028+0000","last_change":"2024-04-26T02:38:59.295670+0000","last_active":"2024-04-26T02:38:59.295028+0000","last_peered":"2024-04-26T02:38:59.295028+0000","last_clean":"2024-04-26T02:23:40.559762+0000","last_became_active":"2024-04-26T02:24:59.607750+0000","last_became_peered":"2024-04-26T02:24:59.607750+0000","last_unstale":"2024-04-26T02:38:59.295028+0000","last_undegraded":"2024-04-26T02:24:59.296484+0000","last_fullsized":"2024-04-26T02:24:59.230081+0000","mapping_epoch":31,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15874,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:42.934335+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:38:42.934335+0000","last_peered":"2024-04-26T02:38:42.934335+0000","last_clean":"2024-04-26T02:38:42.934335+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:38:42.934335+0000","last_undegraded":"2024-04-26T02:38:42.934335+0000","last_fullsized":"2024-04-26T02:38:42.934335+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15880,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T02:38:43.041450+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:38:43.041450+0000","last_peered":"2024-04-26T02:38:43.041450+0000","last_clean":"2024-04-26T02:38:43.041450+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:38:43.041450+0000","last_undegraded":"2024-04-26T02:38:43.041450+0000","last_fullsized":"2024-04-26T02:38:43.041450+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12525,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:00.084687+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:00.084687+0000","last_peered":"2024-04-26T02:39:00.084687+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:00.084687+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26898,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686076,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685764,"statfs":{"total":96636764160,"available":95934222336,"internally_reserved":0,"allocated":503808,"data_stored":192885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591748,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780092,"statfs":{"total":96636764160,"available":96030814208,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:08.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:08.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:08.948 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:08.966 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-04-26T02:39:08.836747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661928,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825432,"statfs":{"total":386547056640,"available":383821242368,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.633467"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":13509,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:05.084177+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:05.084177+0000","last_peered":"2024-04-26T02:39:05.084177+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:05.084177+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1434}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591748,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780092,"statfs":{"total":96636764160,"available":96030814208,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:08.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:09.270 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:09.270 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:09.288 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-04-26T02:39:08.836747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661928,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825432,"statfs":{"total":386547056640,"available":383821242368,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.633467"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":13509,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:05.084177+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:05.084177+0000","last_peered":"2024-04-26T02:39:05.084177+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:05.084177+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1434}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591748,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780092,"statfs":{"total":96636764160,"available":96030814208,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:09.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:09.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 13509, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:39:05.084177+0000', 'last_change': '2024-04-26T02:38:59.298002+0000', 'last_active': '2024-04-26T02:39:05.084177+0000', 'last_peered': '2024-04-26T02:39:05.084177+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:39:05.084177+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 639, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:09.289 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:09.675 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:09.675 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1436/112 objects degraded (1282.143%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":946,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2725814272,"bytes_avail":383821242368,"bytes_total":386547056640,"degraded_objects":1436,"degraded_total":112,"degraded_ratio":12.821428571428571,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":490,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:09.693 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:39:09.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:10.003 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:10.003 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:10.022 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-04-26T02:39:08.836747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661928,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2659697,"kb_avail":374825432,"statfs":{"total":386547056640,"available":383821242368,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2723530291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.633467"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":13509,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:05.084177+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:05.084177+0000","last_peered":"2024-04-26T02:39:05.084177+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:05.084177+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1434}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27388,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591748,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780092,"statfs":{"total":96636764160,"available":96030814208,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215292,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:13.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:13.337 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:13.337 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:13.355 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-04-26T02:39:12.837871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666088,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821272,"statfs":{"total":386547056640,"available":383816982528,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.628499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":14451,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:10.083967+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:10.083967+0000","last_peered":"2024-04-26T02:39:10.083967+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:10.083967+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595908,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775932,"statfs":{"total":96636764160,"available":96026554368,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:13.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:13.661 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:13.661 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:13.680 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-04-26T02:39:12.837871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666088,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821272,"statfs":{"total":386547056640,"available":383816982528,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.628499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":14451,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:10.083967+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:10.083967+0000","last_peered":"2024-04-26T02:39:10.083967+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:10.083967+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595908,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775932,"statfs":{"total":96636764160,"available":96026554368,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:13.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 14451, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:39:10.083967+0000', 'last_change': '2024-04-26T02:38:59.298002+0000', 'last_active': '2024-04-26T02:39:10.083967+0000', 'last_peered': '2024-04-26T02:39:10.083967+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:39:10.083967+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 965, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1110, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:13.682 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:14.073 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:14.074 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 965/112 objects degraded (861.607%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":950,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2730074112,"bytes_avail":383816982528,"bytes_total":386547056640,"degraded_objects":965,"degraded_total":112,"degraded_ratio":8.6160714285714288,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":961,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:14.093 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:39:14.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:14.382 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:14.382 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:14.398 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-04-26T02:39:12.837871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2666088,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2663857,"kb_avail":374821272,"statfs":{"total":386547056640,"available":383816982528,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2727790131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.628499"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":14451,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:10.083967+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:10.083967+0000","last_peered":"2024-04-26T02:39:10.083967+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:10.083967+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27859,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019051,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595908,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93775932,"statfs":{"total":96636764160,"available":96026554368,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215293,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:17.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:17.711 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:17.729 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-04-26T02:39:16.839124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670248,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817112,"statfs":{"total":386547056640,"available":383812722688,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":15563,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:15.086827+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:15.086827+0000","last_peered":"2024-04-26T02:39:15.086827+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:15.086827+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600068,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771772,"statfs":{"total":96636764160,"available":96022294528,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:17.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:18.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:18.063 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-04-26T02:39:16.839124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670248,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817112,"statfs":{"total":386547056640,"available":383812722688,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":15563,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:15.086827+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:15.086827+0000","last_peered":"2024-04-26T02:39:15.086827+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:15.086827+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600068,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771772,"statfs":{"total":96636764160,"available":96022294528,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:18.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:18.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 15563, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-26T02:39:15.086827+0000', 'last_change': '2024-04-26T02:38:59.298002+0000', 'last_active': '2024-04-26T02:39:15.086827+0000', 'last_peered': '2024-04-26T02:39:15.086827+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:39:15.086827+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 31, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 409, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1666, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T02:39:18.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:18.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:18.423 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 409/112 objects degraded (365.179%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":954,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":179368,"bytes_used":2734333952,"bytes_avail":383812722688,"bytes_total":386547056640,"degraded_objects":409,"degraded_total":112,"degraded_ratio":3.6517857142857144,"recovering_objects_per_sec":126,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1517,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:18.440 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-04-26T02:39:18.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:18.727 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:18.745 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-04-26T02:39:18.395606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2670248,"kb_used_data":2160,"kb_used_omap":14,"kb_used_meta":2668017,"kb_avail":374817112,"statfs":{"total":386547056640,"available":383812722688,"internally_reserved":0,"allocated":2211840,"data_stored":927368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2732049971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.559439"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15670,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215084+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:01.215084+0000","last_peered":"2024-04-26T02:39:01.215084+0000","last_clean":"2024-04-26T02:39:01.215084+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:01.215084+0000","last_undegraded":"2024-04-26T02:39:01.215084+0000","last_fullsized":"2024-04-26T02:39:01.215084+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212125+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:01.212125+0000","last_peered":"2024-04-26T02:39:01.212125+0000","last_clean":"2024-04-26T02:39:01.212125+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:01.212125+0000","last_undegraded":"2024-04-26T02:39:01.212125+0000","last_fullsized":"2024-04-26T02:39:01.212125+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15829,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212897+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:01.212897+0000","last_peered":"2024-04-26T02:39:01.212897+0000","last_clean":"2024-04-26T02:39:01.212897+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:01.212897+0000","last_undegraded":"2024-04-26T02:39:01.212897+0000","last_fullsized":"2024-04-26T02:39:01.212897+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15555,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210850+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:01.210850+0000","last_peered":"2024-04-26T02:39:01.210850+0000","last_clean":"2024-04-26T02:39:01.210850+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:01.210850+0000","last_undegraded":"2024-04-26T02:39:01.210850+0000","last_fullsized":"2024-04-26T02:39:01.210850+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215159+0000","last_change":"2024-04-26T02:37:47.162747+0000","last_active":"2024-04-26T02:39:01.215159+0000","last_peered":"2024-04-26T02:39:01.215159+0000","last_clean":"2024-04-26T02:39:01.215159+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:01.215159+0000","last_undegraded":"2024-04-26T02:39:01.215159+0000","last_fullsized":"2024-04-26T02:39:01.215159+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:02.409510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":46381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2015,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15816,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.298381+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:01.298381+0000","last_peered":"2024-04-26T02:39:01.298381+0000","last_clean":"2024-04-26T02:39:01.298381+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:01.298381+0000","last_undegraded":"2024-04-26T02:39:01.298381+0000","last_fullsized":"2024-04-26T02:39:01.298381+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12294,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210959+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:01.210959+0000","last_peered":"2024-04-26T02:39:01.210959+0000","last_clean":"2024-04-26T02:39:01.210959+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:01.210959+0000","last_undegraded":"2024-04-26T02:39:01.210959+0000","last_fullsized":"2024-04-26T02:39:01.210959+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15878,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215492+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:01.215492+0000","last_peered":"2024-04-26T02:39:01.215492+0000","last_clean":"2024-04-26T02:39:01.215492+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:01.215492+0000","last_undegraded":"2024-04-26T02:39:01.215492+0000","last_fullsized":"2024-04-26T02:39:01.215492+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.401616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.212894+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:01.212894+0000","last_peered":"2024-04-26T02:39:01.212894+0000","last_clean":"2024-04-26T02:39:01.212894+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:01.212894+0000","last_undegraded":"2024-04-26T02:39:01.212894+0000","last_fullsized":"2024-04-26T02:39:01.212894+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.149707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15885,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.215088+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:01.215088+0000","last_peered":"2024-04-26T02:39:01.215088+0000","last_clean":"2024-04-26T02:39:01.215088+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:01.215088+0000","last_undegraded":"2024-04-26T02:39:01.215088+0000","last_fullsized":"2024-04-26T02:39:01.215088+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.400300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11853,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213077+0000","last_change":"2024-04-26T02:38:23.400437+0000","last_active":"2024-04-26T02:39:01.213077+0000","last_peered":"2024-04-26T02:39:01.213077+0000","last_clean":"2024-04-26T02:39:01.213077+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:01.213077+0000","last_undegraded":"2024-04-26T02:39:01.213077+0000","last_fullsized":"2024-04-26T02:39:01.213077+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:01.369100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12035,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.210984+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:01.210984+0000","last_peered":"2024-04-26T02:39:01.210984+0000","last_clean":"2024-04-26T02:39:01.210984+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:01.210984+0000","last_undegraded":"2024-04-26T02:39:01.210984+0000","last_fullsized":"2024-04-26T02:39:01.210984+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":15563,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T02:39:15.086827+0000","last_change":"2024-04-26T02:38:59.298002+0000","last_active":"2024-04-26T02:39:15.086827+0000","last_peered":"2024-04-26T02:39:15.086827+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:39:15.086827+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":31,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16114,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211080+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:01.211080+0000","last_peered":"2024-04-26T02:39:01.211080+0000","last_clean":"2024-04-26T02:39:01.211080+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:01.211080+0000","last_undegraded":"2024-04-26T02:39:01.211080+0000","last_fullsized":"2024-04-26T02:39:01.211080+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.211062+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:01.211062+0000","last_peered":"2024-04-26T02:39:01.211062+0000","last_clean":"2024-04-26T02:39:01.211062+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:01.211062+0000","last_undegraded":"2024-04-26T02:39:01.211062+0000","last_fullsized":"2024-04-26T02:39:01.211062+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:03.322850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179368,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187529,"num_write_kb":750211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28415,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686092,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685748,"statfs":{"total":96636764160,"available":95934205952,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019052,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600068,"kb_used_data":636,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771772,"statfs":{"total":96636764160,"available":96022294528,"internally_reserved":0,"allocated":651264,"data_stored":330146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:21.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:22.051 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:22.069 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-04-26T02:39:20.840497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657796,"kb_used_data":2184,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829564,"statfs":{"total":386547056640,"available":383825473536,"internally_reserved":0,"allocated":2236416,"data_stored":936551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19218,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-23188,"num_write_kb":-92764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002625"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15450,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402660+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:18.402660+0000","last_peered":"2024-04-26T02:39:18.402660+0000","last_clean":"2024-04-26T02:39:18.402660+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:18.402660+0000","last_undegraded":"2024-04-26T02:39:18.402660+0000","last_fullsized":"2024-04-26T02:39:18.402660+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:23.179984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15831,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399904+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:18.399904+0000","last_peered":"2024-04-26T02:39:18.399904+0000","last_clean":"2024-04-26T02:39:18.399904+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:18.399904+0000","last_undegraded":"2024-04-26T02:39:18.399904+0000","last_fullsized":"2024-04-26T02:39:18.399904+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.400952+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:18.400952+0000","last_peered":"2024-04-26T02:39:18.400952+0000","last_clean":"2024-04-26T02:39:18.400952+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:18.400952+0000","last_undegraded":"2024-04-26T02:39:18.400952+0000","last_fullsized":"2024-04-26T02:39:18.400952+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12296,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401833+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:18.401833+0000","last_peered":"2024-04-26T02:39:18.401833+0000","last_clean":"2024-04-26T02:39:18.401833+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:18.401833+0000","last_undegraded":"2024-04-26T02:39:18.401833+0000","last_fullsized":"2024-04-26T02:39:18.401833+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15882,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405759+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:19.405759+0000","last_peered":"2024-04-26T02:39:19.405759+0000","last_clean":"2024-04-26T02:39:19.405759+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:19.405759+0000","last_undegraded":"2024-04-26T02:39:19.405759+0000","last_fullsized":"2024-04-26T02:39:19.405759+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.353826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15792,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399992+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:18.399992+0000","last_peered":"2024-04-26T02:39:18.399992+0000","last_clean":"2024-04-26T02:39:18.399992+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:18.399992+0000","last_undegraded":"2024-04-26T02:39:18.399992+0000","last_fullsized":"2024-04-26T02:39:18.399992+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15889,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.404264+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:19.404264+0000","last_peered":"2024-04-26T02:39:19.404264+0000","last_clean":"2024-04-26T02:39:19.404264+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:19.404264+0000","last_undegraded":"2024-04-26T02:39:19.404264+0000","last_fullsized":"2024-04-26T02:39:19.404264+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11863,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.282643+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.282643+0000","last_peered":"2024-04-26T02:39:19.282643+0000","last_clean":"2024-04-26T02:39:19.282643+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.282643+0000","last_undegraded":"2024-04-26T02:39:19.282643+0000","last_fullsized":"2024-04-26T02:39:19.282643+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12037,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401269+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:18.401269+0000","last_peered":"2024-04-26T02:39:18.401269+0000","last_clean":"2024-04-26T02:39:18.401269+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:18.401269+0000","last_undegraded":"2024-04-26T02:39:18.401269+0000","last_fullsized":"2024-04-26T02:39:18.401269+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-26T02:35:52.110862+0000","last_change":"2024-04-26T02:39:19.337290+0000","last_active":"2024-04-26T02:39:19.337290+0000","last_peered":"2024-04-26T02:35:52.110862+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:52.110862+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16116,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401867+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:18.401867+0000","last_peered":"2024-04-26T02:39:18.401867+0000","last_clean":"2024-04-26T02:39:18.401867+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:18.401867+0000","last_undegraded":"2024-04-26T02:39:18.401867+0000","last_fullsized":"2024-04-26T02:39:18.401867+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12070,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402143+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:18.402143+0000","last_peered":"2024-04-26T02:39:18.402143+0000","last_clean":"2024-04-26T02:39:18.402143+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:18.402143+0000","last_undegraded":"2024-04-26T02:39:18.402143+0000","last_fullsized":"2024-04-26T02:39:18.402143+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587600,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784240,"statfs":{"total":96636764160,"available":96035061760,"internally_reserved":0,"allocated":659456,"data_stored":333214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:22.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:22.368 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:22.369 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:22.385 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-04-26T02:39:20.840497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657796,"kb_used_data":2184,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829564,"statfs":{"total":386547056640,"available":383825473536,"internally_reserved":0,"allocated":2236416,"data_stored":936551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19218,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-23188,"num_write_kb":-92764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002625"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15450,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402660+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:18.402660+0000","last_peered":"2024-04-26T02:39:18.402660+0000","last_clean":"2024-04-26T02:39:18.402660+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:18.402660+0000","last_undegraded":"2024-04-26T02:39:18.402660+0000","last_fullsized":"2024-04-26T02:39:18.402660+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:23.179984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15831,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399904+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:18.399904+0000","last_peered":"2024-04-26T02:39:18.399904+0000","last_clean":"2024-04-26T02:39:18.399904+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:18.399904+0000","last_undegraded":"2024-04-26T02:39:18.399904+0000","last_fullsized":"2024-04-26T02:39:18.399904+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.400952+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:18.400952+0000","last_peered":"2024-04-26T02:39:18.400952+0000","last_clean":"2024-04-26T02:39:18.400952+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:18.400952+0000","last_undegraded":"2024-04-26T02:39:18.400952+0000","last_fullsized":"2024-04-26T02:39:18.400952+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16060,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-26T02:39:01.213597+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:01.213597+0000","last_peered":"2024-04-26T02:39:01.213597+0000","last_clean":"2024-04-26T02:39:01.213597+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:01.213597+0000","last_undegraded":"2024-04-26T02:39:01.213597+0000","last_fullsized":"2024-04-26T02:39:01.213597+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:05.640734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12296,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401833+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:18.401833+0000","last_peered":"2024-04-26T02:39:18.401833+0000","last_clean":"2024-04-26T02:39:18.401833+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:18.401833+0000","last_undegraded":"2024-04-26T02:39:18.401833+0000","last_fullsized":"2024-04-26T02:39:18.401833+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15882,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405759+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:19.405759+0000","last_peered":"2024-04-26T02:39:19.405759+0000","last_clean":"2024-04-26T02:39:19.405759+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:19.405759+0000","last_undegraded":"2024-04-26T02:39:19.405759+0000","last_fullsized":"2024-04-26T02:39:19.405759+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.353826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15792,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399992+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:18.399992+0000","last_peered":"2024-04-26T02:39:18.399992+0000","last_clean":"2024-04-26T02:39:18.399992+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:18.399992+0000","last_undegraded":"2024-04-26T02:39:18.399992+0000","last_fullsized":"2024-04-26T02:39:18.399992+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15889,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.404264+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:19.404264+0000","last_peered":"2024-04-26T02:39:19.404264+0000","last_clean":"2024-04-26T02:39:19.404264+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:19.404264+0000","last_undegraded":"2024-04-26T02:39:19.404264+0000","last_fullsized":"2024-04-26T02:39:19.404264+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11863,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.282643+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.282643+0000","last_peered":"2024-04-26T02:39:19.282643+0000","last_clean":"2024-04-26T02:39:19.282643+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.282643+0000","last_undegraded":"2024-04-26T02:39:19.282643+0000","last_fullsized":"2024-04-26T02:39:19.282643+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12037,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401269+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:18.401269+0000","last_peered":"2024-04-26T02:39:18.401269+0000","last_clean":"2024-04-26T02:39:18.401269+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:18.401269+0000","last_undegraded":"2024-04-26T02:39:18.401269+0000","last_fullsized":"2024-04-26T02:39:18.401269+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-26T02:35:52.110862+0000","last_change":"2024-04-26T02:39:19.337290+0000","last_active":"2024-04-26T02:39:19.337290+0000","last_peered":"2024-04-26T02:35:52.110862+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:52.110862+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16116,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401867+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:18.401867+0000","last_peered":"2024-04-26T02:39:18.401867+0000","last_clean":"2024-04-26T02:39:18.401867+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:18.401867+0000","last_undegraded":"2024-04-26T02:39:18.401867+0000","last_fullsized":"2024-04-26T02:39:18.401867+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12070,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402143+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:18.402143+0000","last_peered":"2024-04-26T02:39:18.402143+0000","last_clean":"2024-04-26T02:39:18.402143+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:18.402143+0000","last_undegraded":"2024-04-26T02:39:18.402143+0000","last_fullsized":"2024-04-26T02:39:18.402143+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587600,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784240,"statfs":{"total":96636764160,"available":96035061760,"internally_reserved":0,"allocated":659456,"data_stored":333214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694348,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677492,"statfs":{"total":96636764160,"available":95925751808,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:22.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-04-26T02:39:22.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "40'12120", 'reported_seq': 12377, 'reported_epoch': 63, 'state': 'peering', 'last_fresh': '2024-04-26T02:35:52.110862+0000', 'last_change': '2024-04-26T02:39:19.337290+0000', 'last_active': '2024-04-26T02:39:19.337290+0000', 'last_peered': '2024-04-26T02:35:52.110862+0000', 'last_clean': '2024-04-26T02:23:50.679937+0000', 'last_became_active': '2024-04-26T02:24:59.619013+0000', 'last_became_peered': '2024-04-26T02:24:59.619013+0000', 'last_unstale': '2024-04-26T02:35:52.110862+0000', 'last_undegraded': '2024-04-26T02:24:59.301239+0000', 'last_fullsized': '2024-04-26T02:24:59.230890+0000', 'mapping_epoch': 63, 'log_start': "32'2100", 'ondisk_log_start': "32'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'last_clean_scrub_stamp': '2024-04-26T02:23:40.559762+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12118, 'num_write_kb': 48484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [1, 3, 2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T02:39:22.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T02:39:22.735 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:22.735 INFO:teuthology.orchestra.run.smithi177.stdout:{"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":959,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":64,"num_osds":4,"num_up_osds":4,"osd_up_since":1714098297,"num_in_osds":4,"osd_in_since":1714098204,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25,"data_bytes":160150,"bytes_used":2721583104,"bytes_avail":383825473536,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T02:38:34.826471+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T02:39:22.753 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T02:39:22.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:23.045 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:23.045 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:23.064 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-04-26T02:39:22.841061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657804,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829556,"statfs":{"total":386547056640,"available":383825465344,"internally_reserved":0,"allocated":2252800,"data_stored":942666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19218,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-23188,"num_write_kb":-92764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002620"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15450,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402660+0000","last_change":"2024-04-26T02:35:56.049250+0000","last_active":"2024-04-26T02:39:18.402660+0000","last_peered":"2024-04-26T02:39:18.402660+0000","last_clean":"2024-04-26T02:39:18.402660+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:18.402660+0000","last_undegraded":"2024-04-26T02:39:18.402660+0000","last_fullsized":"2024-04-26T02:39:18.402660+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:23.179984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15831,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399904+0000","last_change":"2024-04-26T02:36:12.099001+0000","last_active":"2024-04-26T02:39:18.399904+0000","last_peered":"2024-04-26T02:39:18.399904+0000","last_clean":"2024-04-26T02:39:18.399904+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:18.399904+0000","last_undegraded":"2024-04-26T02:39:18.399904+0000","last_fullsized":"2024-04-26T02:39:18.399904+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11629,"num_write_kb":46511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.400952+0000","last_change":"2024-04-26T02:30:41.964394+0000","last_active":"2024-04-26T02:39:18.400952+0000","last_peered":"2024-04-26T02:39:18.400952+0000","last_clean":"2024-04-26T02:39:18.400952+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:18.400952+0000","last_undegraded":"2024-04-26T02:39:18.400952+0000","last_fullsized":"2024-04-26T02:39:18.400952+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":12034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16064,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.400563+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:19.400563+0000","last_peered":"2024-04-26T02:39:19.400563+0000","last_clean":"2024-04-26T02:39:19.400563+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:19.400563+0000","last_undegraded":"2024-04-26T02:39:19.400563+0000","last_fullsized":"2024-04-26T02:39:19.400563+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.572949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12296,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401833+0000","last_change":"2024-04-26T02:36:32.213371+0000","last_active":"2024-04-26T02:39:18.401833+0000","last_peered":"2024-04-26T02:39:18.401833+0000","last_clean":"2024-04-26T02:39:18.401833+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:18.401833+0000","last_undegraded":"2024-04-26T02:39:18.401833+0000","last_fullsized":"2024-04-26T02:39:18.401833+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15882,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405759+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:19.405759+0000","last_peered":"2024-04-26T02:39:19.405759+0000","last_clean":"2024-04-26T02:39:19.405759+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:19.405759+0000","last_undegraded":"2024-04-26T02:39:19.405759+0000","last_fullsized":"2024-04-26T02:39:19.405759+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.353826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15792,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.399992+0000","last_change":"2024-04-26T02:38:04.315720+0000","last_active":"2024-04-26T02:39:18.399992+0000","last_peered":"2024-04-26T02:39:18.399992+0000","last_clean":"2024-04-26T02:39:18.399992+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:18.399992+0000","last_undegraded":"2024-04-26T02:39:18.399992+0000","last_fullsized":"2024-04-26T02:39:18.399992+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.094667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15889,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.404264+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:19.404264+0000","last_peered":"2024-04-26T02:39:19.404264+0000","last_clean":"2024-04-26T02:39:19.404264+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:19.404264+0000","last_undegraded":"2024-04-26T02:39:19.404264+0000","last_fullsized":"2024-04-26T02:39:19.404264+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11863,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.282643+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.282643+0000","last_peered":"2024-04-26T02:39:19.282643+0000","last_clean":"2024-04-26T02:39:19.282643+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.282643+0000","last_undegraded":"2024-04-26T02:39:19.282643+0000","last_fullsized":"2024-04-26T02:39:19.282643+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12037,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401269+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:18.401269+0000","last_peered":"2024-04-26T02:39:18.401269+0000","last_clean":"2024-04-26T02:39:18.401269+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:18.401269+0000","last_undegraded":"2024-04-26T02:39:18.401269+0000","last_fullsized":"2024-04-26T02:39:18.401269+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12377,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-26T02:35:52.110862+0000","last_change":"2024-04-26T02:39:19.337290+0000","last_active":"2024-04-26T02:39:19.337290+0000","last_peered":"2024-04-26T02:35:52.110862+0000","last_clean":"2024-04-26T02:23:50.679937+0000","last_became_active":"2024-04-26T02:24:59.619013+0000","last_became_peered":"2024-04-26T02:24:59.619013+0000","last_unstale":"2024-04-26T02:35:52.110862+0000","last_undegraded":"2024-04-26T02:24:59.301239+0000","last_fullsized":"2024-04-26T02:24:59.230890+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16116,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.401867+0000","last_change":"2024-04-26T02:28:52.824678+0000","last_active":"2024-04-26T02:39:18.401867+0000","last_peered":"2024-04-26T02:39:18.401867+0000","last_clean":"2024-04-26T02:39:18.401867+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:18.401867+0000","last_undegraded":"2024-04-26T02:39:18.401867+0000","last_fullsized":"2024-04-26T02:39:18.401867+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":47413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":16818176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12070,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-26T02:39:18.402143+0000","last_change":"2024-04-26T02:37:06.992616+0000","last_active":"2024-04-26T02:39:18.402143+0000","last_peered":"2024-04-26T02:39:18.402143+0000","last_clean":"2024-04-26T02:39:18.402143+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:18.402143+0000","last_undegraded":"2024-04-26T02:39:18.402143+0000","last_fullsized":"2024-04-26T02:39:18.402143+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":160150,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164341,"num_write_kb":657447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24737,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689740,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682100,"statfs":{"total":96636764160,"available":95930470400,"internally_reserved":0,"allocated":520192,"data_stored":199074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019053,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587600,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784240,"statfs":{"total":96636764160,"available":96035061760,"internally_reserved":0,"allocated":659456,"data_stored":333214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:26.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:26.375 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:26.388 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2024-04-26T02:39:24.841596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89684,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94012,"num_write_kb":376096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12416,"num_bytes_recovered":17481728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657816,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829544,"statfs":{"total":386547056640,"available":383825453056,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-89684,"num_objects":-14,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-93517,"num_write_kb":-374115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15999,"num_bytes_recovered":-28852224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002472"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16064,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.400563+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:19.400563+0000","last_peered":"2024-04-26T02:39:19.400563+0000","last_clean":"2024-04-26T02:39:19.400563+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:19.400563+0000","last_undegraded":"2024-04-26T02:39:19.400563+0000","last_fullsized":"2024-04-26T02:39:19.400563+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.572949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15882,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405759+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:19.405759+0000","last_peered":"2024-04-26T02:39:19.405759+0000","last_clean":"2024-04-26T02:39:19.405759+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:19.405759+0000","last_undegraded":"2024-04-26T02:39:19.405759+0000","last_fullsized":"2024-04-26T02:39:19.405759+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.353826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15889,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.404264+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:19.404264+0000","last_peered":"2024-04-26T02:39:19.404264+0000","last_clean":"2024-04-26T02:39:19.404264+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:19.404264+0000","last_undegraded":"2024-04-26T02:39:19.404264+0000","last_fullsized":"2024-04-26T02:39:19.404264+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":89684,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94012,"num_write_kb":376096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12416,"num_bytes_recovered":17481728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:26.390 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-04-26T02:39:26.390 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-04-26T02:39:26.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:26.688 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:26.688 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:26.706 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2024-04-26T02:39:24.841596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89684,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94012,"num_write_kb":376096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12416,"num_bytes_recovered":17481728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657816,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829544,"statfs":{"total":386547056640,"available":383825453056,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-89684,"num_objects":-14,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-93517,"num_write_kb":-374115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15999,"num_bytes_recovered":-28852224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002472"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16064,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.400563+0000","last_change":"2024-04-26T02:26:45.797573+0000","last_active":"2024-04-26T02:39:19.400563+0000","last_peered":"2024-04-26T02:39:19.400563+0000","last_clean":"2024-04-26T02:39:19.400563+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:19.400563+0000","last_undegraded":"2024-04-26T02:39:19.400563+0000","last_fullsized":"2024-04-26T02:39:19.400563+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.572949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":17055744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15882,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405759+0000","last_change":"2024-04-26T02:33:24.459075+0000","last_active":"2024-04-26T02:39:19.405759+0000","last_peered":"2024-04-26T02:39:19.405759+0000","last_clean":"2024-04-26T02:39:19.405759+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:19.405759+0000","last_undegraded":"2024-04-26T02:39:19.405759+0000","last_fullsized":"2024-04-26T02:39:19.405759+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.353826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":46945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15889,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.404264+0000","last_change":"2024-04-26T02:38:43.041450+0000","last_active":"2024-04-26T02:39:19.404264+0000","last_peered":"2024-04-26T02:39:19.404264+0000","last_clean":"2024-04-26T02:39:19.404264+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:19.404264+0000","last_undegraded":"2024-04-26T02:39:19.404264+0000","last_fullsized":"2024-04-26T02:39:19.404264+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:22.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19218,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":46860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":89684,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94012,"num_write_kb":376096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12416,"num_bytes_recovered":17481728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686108,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685732,"statfs":{"total":96636764160,"available":95934189568,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:26.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:26.990 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:26.990 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:27.004 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-04-26T02:39:26.842082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-140932,"num_objects":-22,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128717,"num_write_kb":-514940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22118,"num_bytes_recovered":-46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.446475"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16072,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.665135+0000","last_change":"2024-04-26T02:39:23.665180+0000","last_active":"2024-04-26T02:39:23.665135+0000","last_peered":"2024-04-26T02:39:23.665135+0000","last_clean":"2024-04-26T02:39:23.665135+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:23.665135+0000","last_undegraded":"2024-04-26T02:39:23.665135+0000","last_fullsized":"2024-04-26T02:39:23.665135+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:23.665014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.665014+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:11:08.402573+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.475283+0000","last_change":"2024-04-26T02:39:20.475318+0000","last_active":"2024-04-26T02:39:20.475283+0000","last_peered":"2024-04-26T02:39:20.475283+0000","last_clean":"2024-04-26T02:39:20.475283+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:20.475283+0000","last_undegraded":"2024-04-26T02:39:20.475283+0000","last_fullsized":"2024-04-26T02:39:20.475283+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:20.475175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.475175+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:57.691335+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15897,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.419930+0000","last_change":"2024-04-26T02:39:23.419995+0000","last_active":"2024-04-26T02:39:23.419930+0000","last_peered":"2024-04-26T02:39:23.419930+0000","last_clean":"2024-04-26T02:39:23.419930+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:23.419930+0000","last_undegraded":"2024-04-26T02:39:23.419930+0000","last_fullsized":"2024-04-26T02:39:23.419930+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:23.419747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.419747+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:36:17.889251+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:27.005 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-04-26T02:39:27.005 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T02:39:27.024 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T02:39:27.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:27.302 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:27.303 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:27.319 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-04-26T02:39:26.842082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-140932,"num_objects":-22,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128717,"num_write_kb":-514940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22118,"num_bytes_recovered":-46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.446475"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16072,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.665135+0000","last_change":"2024-04-26T02:39:23.665180+0000","last_active":"2024-04-26T02:39:23.665135+0000","last_peered":"2024-04-26T02:39:23.665135+0000","last_clean":"2024-04-26T02:39:23.665135+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:23.665135+0000","last_undegraded":"2024-04-26T02:39:23.665135+0000","last_fullsized":"2024-04-26T02:39:23.665135+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:23.665014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.665014+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:11:08.402573+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.475283+0000","last_change":"2024-04-26T02:39:20.475318+0000","last_active":"2024-04-26T02:39:20.475283+0000","last_peered":"2024-04-26T02:39:20.475283+0000","last_clean":"2024-04-26T02:39:20.475283+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:20.475283+0000","last_undegraded":"2024-04-26T02:39:20.475283+0000","last_fullsized":"2024-04-26T02:39:20.475283+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:20.475175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.475175+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:57.691335+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15897,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.419930+0000","last_change":"2024-04-26T02:39:23.419995+0000","last_active":"2024-04-26T02:39:23.419930+0000","last_peered":"2024-04-26T02:39:23.419930+0000","last_clean":"2024-04-26T02:39:23.419930+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:23.419930+0000","last_undegraded":"2024-04-26T02:39:23.419930+0000","last_fullsized":"2024-04-26T02:39:23.419930+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:23.419747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.419747+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:36:17.889251+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:27.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:27.586 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:27.587 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:27.606 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-04-26T02:39:26.842082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-140932,"num_objects":-22,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128717,"num_write_kb":-514940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22118,"num_bytes_recovered":-46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.446475"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16072,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.665135+0000","last_change":"2024-04-26T02:39:23.665180+0000","last_active":"2024-04-26T02:39:23.665135+0000","last_peered":"2024-04-26T02:39:23.665135+0000","last_clean":"2024-04-26T02:39:23.665135+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:23.665135+0000","last_undegraded":"2024-04-26T02:39:23.665135+0000","last_fullsized":"2024-04-26T02:39:23.665135+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:23.665014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.665014+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:11:08.402573+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.475283+0000","last_change":"2024-04-26T02:39:20.475318+0000","last_active":"2024-04-26T02:39:20.475283+0000","last_peered":"2024-04-26T02:39:20.475283+0000","last_clean":"2024-04-26T02:39:20.475283+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:20.475283+0000","last_undegraded":"2024-04-26T02:39:20.475283+0000","last_fullsized":"2024-04-26T02:39:20.475283+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:20.475175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.475175+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:57.691335+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15897,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.419930+0000","last_change":"2024-04-26T02:39:23.419995+0000","last_active":"2024-04-26T02:39:23.419930+0000","last_peered":"2024-04-26T02:39:23.419930+0000","last_clean":"2024-04-26T02:39:23.419930+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:23.419930+0000","last_undegraded":"2024-04-26T02:39:23.419930+0000","last_fullsized":"2024-04-26T02:39:23.419930+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:23.419747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.419747+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:36:17.889251+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:27.607 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T02:39:27.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:27.881 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:27.881 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:27.896 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-04-26T02:39:26.842082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-140932,"num_objects":-22,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128717,"num_write_kb":-514940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22118,"num_bytes_recovered":-46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.446475"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16072,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.665135+0000","last_change":"2024-04-26T02:39:23.665180+0000","last_active":"2024-04-26T02:39:23.665135+0000","last_peered":"2024-04-26T02:39:23.665135+0000","last_clean":"2024-04-26T02:39:23.665135+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:23.665135+0000","last_undegraded":"2024-04-26T02:39:23.665135+0000","last_fullsized":"2024-04-26T02:39:23.665135+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:23.665014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.665014+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:11:08.402573+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.475283+0000","last_change":"2024-04-26T02:39:20.475318+0000","last_active":"2024-04-26T02:39:20.475283+0000","last_peered":"2024-04-26T02:39:20.475283+0000","last_clean":"2024-04-26T02:39:20.475283+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:20.475283+0000","last_undegraded":"2024-04-26T02:39:20.475283+0000","last_fullsized":"2024-04-26T02:39:20.475283+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:20.475175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.475175+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:57.691335+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15897,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.419930+0000","last_change":"2024-04-26T02:39:23.419995+0000","last_active":"2024-04-26T02:39:23.419930+0000","last_peered":"2024-04-26T02:39:23.419930+0000","last_clean":"2024-04-26T02:39:23.419930+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:23.419930+0000","last_undegraded":"2024-04-26T02:39:23.419930+0000","last_fullsized":"2024-04-26T02:39:23.419930+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:23.419747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.419747+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:36:17.889251+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12039,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.397912+0000","last_change":"2024-04-26T02:37:29.048442+0000","last_active":"2024-04-26T02:39:19.397912+0000","last_peered":"2024-04-26T02:39:19.397912+0000","last_clean":"2024-04-26T02:39:19.397912+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:19.397912+0000","last_undegraded":"2024-04-26T02:39:19.397912+0000","last_fullsized":"2024-04-26T02:39:19.397912+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:20.121135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12387,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.511418+0000","last_change":"2024-04-26T02:39:19.511418+0000","last_active":"2024-04-26T02:39:19.511418+0000","last_peered":"2024-04-26T02:39:19.511418+0000","last_clean":"2024-04-26T02:39:19.511418+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:19.511418+0000","last_undegraded":"2024-04-26T02:39:19.511418+0000","last_fullsized":"2024-04-26T02:39:19.511418+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12118,"num_write_kb":48484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12076,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T02:39:24.295842+0000","last_change":"2024-04-26T02:39:24.292568+0000","last_active":"2024-04-26T02:39:24.295842+0000","last_peered":"2024-04-26T02:39:24.295842+0000","last_clean":"2024-04-26T02:39:24.295842+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.295842+0000","last_undegraded":"2024-04-26T02:39:24.295842+0000","last_fullsized":"2024-04-26T02:39:24.295842+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":47180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38436,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58812,"num_write_kb":235271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019054,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:27.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:39:28.174 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:28.174 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":64,"fsid":"13088462-cff2-40f2-960d-bd23a2ee33c5","created":"2024-04-26T02:23:23.478039+0000","modified":"2024-04-26T02:39:19.390138+0000","last_up_change":"2024-04-26T02:24:57.516281+0000","last_in_change":"2024-04-26T02:23:24.853353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-04-26T02:23:39.604824+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"77e13700-dc07-44ef-92cd-f9e7b3b12087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":24,"up_thru":48,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":2332209013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":2332209013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":2332209013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":2332209013}]},"public_addr":"172.21.15.177:6803/2332209013","cluster_addr":"172.21.15.177:6805/2332209013","heartbeat_back_addr":"172.21.15.177:6807/2332209013","heartbeat_front_addr":"172.21.15.177:6806/2332209013","state":["exists","up"]},{"osd":1,"uuid":"bc6b2040-3b7c-410f-b7da-a33b88edf242","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":30,"up_thru":63,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3273495033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":3273495033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":3273495033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":3273495033}]},"public_addr":"172.21.15.177:6812/3273495033","cluster_addr":"172.21.15.177:6813/3273495033","heartbeat_back_addr":"172.21.15.177:6815/3273495033","heartbeat_front_addr":"172.21.15.177:6814/3273495033","state":["exists","up"]},{"osd":2,"uuid":"1bcd23e3-1de4-494e-b73d-51c88edda7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":3328945250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3328945250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3328945250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3328945250}]},"public_addr":"172.21.15.177:6800/3328945250","cluster_addr":"172.21.15.177:6801/3328945250","heartbeat_back_addr":"172.21.15.177:6804/3328945250","heartbeat_front_addr":"172.21.15.177:6802/3328945250","state":["exists","up"]},{"osd":3,"uuid":"51cee477-d535-4ba9-bee6-4288e817a9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":27,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":252562504}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":252562504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":252562504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":252562504}]},"public_addr":"172.21.15.177:6808/252562504","cluster_addr":"172.21.15.177:6809/252562504","heartbeat_back_addr":"172.21.15.177:6811/252562504","heartbeat_front_addr":"172.21.15.177:6810/252562504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.370664+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.119168+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.161391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:23:30.172138+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:39:29.192 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T02:39:29.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:39:29.348 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:39:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:39:29.349 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:39:29.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T02:39:29.632 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T02:39:29.648 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T02:39:29.649 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:39:29.791 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:39:29.791 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:39:29.791 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:39:29.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T02:39:30.088 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T02:39:30.120 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T02:39:30.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:39:30.253 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:39:30.253 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:39:30.253 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:39:30.267 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T02:39:30.544 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T02:39:30.565 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T02:39:30.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:39:30.710 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-26T02:39:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:39:30.711 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-26T02:39:30.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T02:39:30.992 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T02:39:31.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:31.305 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:31.305 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:31.321 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-04-26T02:39:30.842909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23124,"num_write_kb":92496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-166556,"num_objects":-26,"num_object_clones":0,"num_object_copies":-104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-152810,"num_write_kb":-611332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22684,"num_bytes_recovered":-46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.448660"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15674,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.403985+0000","last_change":"2024-04-26T02:36:49.399864+0000","last_active":"2024-04-26T02:39:19.403985+0000","last_peered":"2024-04-26T02:39:19.403985+0000","last_clean":"2024-04-26T02:39:19.403985+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:19.403985+0000","last_undegraded":"2024-04-26T02:39:19.403985+0000","last_fullsized":"2024-04-26T02:39:19.403985+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:31:51.662574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15460,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.376558+0000","last_change":"2024-04-26T02:39:23.376614+0000","last_active":"2024-04-26T02:39:23.376558+0000","last_peered":"2024-04-26T02:39:23.376558+0000","last_clean":"2024-04-26T02:39:23.376558+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:23.376558+0000","last_undegraded":"2024-04-26T02:39:23.376558+0000","last_fullsized":"2024-04-26T02:39:23.376558+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:23.376423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.376423+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:42.700512+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15841,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.095889+0000","last_change":"2024-04-26T02:39:21.095928+0000","last_active":"2024-04-26T02:39:21.095889+0000","last_peered":"2024-04-26T02:39:21.095889+0000","last_clean":"2024-04-26T02:39:21.095889+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:21.095889+0000","last_undegraded":"2024-04-26T02:39:21.095889+0000","last_fullsized":"2024-04-26T02:39:21.095889+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:21.095760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.095760+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:31:34.951342+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15567,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:21.375416+0000","last_change":"2024-04-26T02:39:21.375457+0000","last_active":"2024-04-26T02:39:21.375416+0000","last_peered":"2024-04-26T02:39:21.375416+0000","last_clean":"2024-04-26T02:39:21.375416+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:21.375416+0000","last_undegraded":"2024-04-26T02:39:21.375416+0000","last_fullsized":"2024-04-26T02:39:21.375416+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:21.375272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:21.375272+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:43.436222+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15760,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.583332+0000","last_change":"2024-04-26T02:39:19.583387+0000","last_active":"2024-04-26T02:39:19.583332+0000","last_peered":"2024-04-26T02:39:19.583332+0000","last_clean":"2024-04-26T02:39:19.583332+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:19.583332+0000","last_undegraded":"2024-04-26T02:39:19.583332+0000","last_fullsized":"2024-04-26T02:39:19.583332+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:19.583181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.583181+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:28.219232+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15820,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.405967+0000","last_change":"2024-04-26T02:39:01.298381+0000","last_active":"2024-04-26T02:39:19.405967+0000","last_peered":"2024-04-26T02:39:19.405967+0000","last_clean":"2024-04-26T02:39:19.405967+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:19.405967+0000","last_undegraded":"2024-04-26T02:39:19.405967+0000","last_fullsized":"2024-04-26T02:39:19.405967+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:23:40.559762+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:10:09.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16072,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.665135+0000","last_change":"2024-04-26T02:39:23.665180+0000","last_active":"2024-04-26T02:39:23.665135+0000","last_peered":"2024-04-26T02:39:23.665135+0000","last_clean":"2024-04-26T02:39:23.665135+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:23.665135+0000","last_undegraded":"2024-04-26T02:39:23.665135+0000","last_fullsized":"2024-04-26T02:39:23.665135+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:23.665014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.665014+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:11:08.402573+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.313618+0000","last_change":"2024-04-26T02:39:22.313643+0000","last_active":"2024-04-26T02:39:22.313618+0000","last_peered":"2024-04-26T02:39:22.313618+0000","last_clean":"2024-04-26T02:39:22.313618+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:22.313618+0000","last_undegraded":"2024-04-26T02:39:22.313618+0000","last_fullsized":"2024-04-26T02:39:22.313618+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:22.313525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.313525+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:03:17.296233+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.475283+0000","last_change":"2024-04-26T02:39:20.475318+0000","last_active":"2024-04-26T02:39:20.475283+0000","last_peered":"2024-04-26T02:39:20.475283+0000","last_clean":"2024-04-26T02:39:20.475283+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:20.475283+0000","last_undegraded":"2024-04-26T02:39:20.475283+0000","last_fullsized":"2024-04-26T02:39:20.475283+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:20.475175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.475175+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:57.691335+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15802,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:22.105522+0000","last_change":"2024-04-26T02:39:22.105590+0000","last_active":"2024-04-26T02:39:22.105522+0000","last_peered":"2024-04-26T02:39:22.105522+0000","last_clean":"2024-04-26T02:39:22.105522+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:22.105522+0000","last_undegraded":"2024-04-26T02:39:22.105522+0000","last_fullsized":"2024-04-26T02:39:22.105522+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:22.105336+0000","last_clean_scrub_stamp":"2024-04-26T02:39:22.105336+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:24.653554+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15897,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:23.419930+0000","last_change":"2024-04-26T02:39:23.419995+0000","last_active":"2024-04-26T02:39:23.419930+0000","last_peered":"2024-04-26T02:39:23.419930+0000","last_clean":"2024-04-26T02:39:23.419930+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:23.419930+0000","last_undegraded":"2024-04-26T02:39:23.419930+0000","last_fullsized":"2024-04-26T02:39:23.419930+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:23.419747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:23.419747+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:36:17.889251+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11865,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:19.399234+0000","last_change":"2024-04-26T02:39:19.282692+0000","last_active":"2024-04-26T02:39:19.399234+0000","last_peered":"2024-04-26T02:39:19.399234+0000","last_clean":"2024-04-26T02:39:19.399234+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:19.399234+0000","last_undegraded":"2024-04-26T02:39:19.399234+0000","last_fullsized":"2024-04-26T02:39:19.399234+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:19.282452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:19.282452+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:22:36.332651+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12047,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:25.458050+0000","last_change":"2024-04-26T02:39:25.458091+0000","last_active":"2024-04-26T02:39:25.458050+0000","last_peered":"2024-04-26T02:39:25.458050+0000","last_clean":"2024-04-26T02:39:25.458050+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:25.458050+0000","last_undegraded":"2024-04-26T02:39:25.458050+0000","last_fullsized":"2024-04-26T02:39:25.458050+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11778","last_scrub_stamp":"2024-04-26T02:39:25.457946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:25.457946+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:04:28.222174+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12395,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:26.463000+0000","last_change":"2024-04-26T02:39:26.463039+0000","last_active":"2024-04-26T02:39:26.463000+0000","last_peered":"2024-04-26T02:39:26.463000+0000","last_clean":"2024-04-26T02:39:26.463000+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:26.463000+0000","last_undegraded":"2024-04-26T02:39:26.463000+0000","last_fullsized":"2024-04-26T02:39:26.463000+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12120","last_scrub_stamp":"2024-04-26T02:39:26.462902+0000","last_deep_scrub":"40'12120","last_deep_scrub_stamp":"2024-04-26T02:39:26.462902+0000","last_clean_scrub_stamp":"2024-04-26T02:39:26.462902+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:52:25.269477+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16126,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:20.317848+0000","last_change":"2024-04-26T02:39:20.317915+0000","last_active":"2024-04-26T02:39:20.317848+0000","last_peered":"2024-04-26T02:39:20.317848+0000","last_clean":"2024-04-26T02:39:20.317848+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:20.317848+0000","last_undegraded":"2024-04-26T02:39:20.317848+0000","last_fullsized":"2024-04-26T02:39:20.317848+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:20.317658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:20.317658+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:03:18.717614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12081,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:24.373939+0000","last_change":"2024-04-26T02:39:24.373975+0000","last_active":"2024-04-26T02:39:24.373939+0000","last_peered":"2024-04-26T02:39:24.373939+0000","last_clean":"2024-04-26T02:39:24.373939+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:24.373939+0000","last_undegraded":"2024-04-26T02:39:24.373939+0000","last_fullsized":"2024-04-26T02:39:24.373939+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11795","last_scrub_stamp":"2024-04-26T02:39:24.373813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:23:40.559762+0000","last_clean_scrub_stamp":"2024-04-26T02:39:24.373813+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:28.460540+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23124,"num_write_kb":92496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:31.325 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-04-26T02:23:40.559762+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=23, tm_sec=40, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.325 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-04-26T02:39:23.376423+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=23, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.325 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-04-26T02:39:21.095760+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=21, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.325 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T02:39:21.375272+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=21, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-26T02:39:19.583181+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=19, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-04-26T02:23:40.559762+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=23, tm_sec=40, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-26T02:39:23.665014+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=23, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-26T02:39:22.313525+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=22, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-26T02:39:20.475175+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-26T02:39:22.105336+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=22, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-26T02:39:23.419747+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=23, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.326 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-26T02:39:19.282452+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=19, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.327 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-04-26T02:39:25.457946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.327 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-04-26T02:39:26.462902+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=26, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.327 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-04-26T02:39:20.317658+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.327 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-04-26T02:39:24.373813+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=24, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=39, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:39:31.327 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T02:39:51.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:39:51.632 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-26T02:39:51.632 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-04-26T02:39:51.649 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2024-04-26T02:39:50.848422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23124,"num_write_kb":92496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2657808,"kb_used_data":2224,"kb_used_omap":14,"kb_used_meta":2655537,"kb_avail":374829552,"statfs":{"total":386547056640,"available":383825461248,"internally_reserved":0,"allocated":2277376,"data_stored":951828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14797,"internal_metadata":2719270451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003638"},"pg_stats":[{"pgid":"1.f","version":"40'11524","reported_seq":15683,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:32.424741+0000","last_change":"2024-04-26T02:39:32.424785+0000","last_active":"2024-04-26T02:39:32.424741+0000","last_peered":"2024-04-26T02:39:32.424741+0000","last_clean":"2024-04-26T02:39:32.424741+0000","last_became_active":"2024-04-26T02:36:49.322433+0000","last_became_peered":"2024-04-26T02:36:49.322433+0000","last_unstale":"2024-04-26T02:39:32.424741+0000","last_undegraded":"2024-04-26T02:39:32.424741+0000","last_fullsized":"2024-04-26T02:39:32.424741+0000","mapping_epoch":47,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11524","last_scrub_stamp":"2024-04-26T02:39:32.424575+0000","last_deep_scrub":"40'11524","last_deep_scrub_stamp":"2024-04-26T02:39:32.424575+0000","last_clean_scrub_stamp":"2024-04-26T02:39:32.424575+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:43:32.217866+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":46096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'11830","reported_seq":15469,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:32.456982+0000","last_change":"2024-04-26T02:39:32.457009+0000","last_active":"2024-04-26T02:39:32.456982+0000","last_peered":"2024-04-26T02:39:32.456982+0000","last_clean":"2024-04-26T02:39:32.456982+0000","last_became_active":"2024-04-26T02:35:55.930435+0000","last_became_peered":"2024-04-26T02:35:55.930435+0000","last_unstale":"2024-04-26T02:39:32.456982+0000","last_undegraded":"2024-04-26T02:39:32.456982+0000","last_fullsized":"2024-04-26T02:39:32.456982+0000","mapping_epoch":41,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11830","last_scrub_stamp":"2024-04-26T02:39:32.456877+0000","last_deep_scrub":"40'11830","last_deep_scrub_stamp":"2024-04-26T02:39:32.456877+0000","last_clean_scrub_stamp":"2024-04-26T02:39:32.456877+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:41.969183+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'11630","reported_seq":15851,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:31.212798+0000","last_change":"2024-04-26T02:39:31.212798+0000","last_active":"2024-04-26T02:39:31.212798+0000","last_peered":"2024-04-26T02:39:31.212798+0000","last_clean":"2024-04-26T02:39:31.212798+0000","last_became_active":"2024-04-26T02:36:12.011398+0000","last_became_peered":"2024-04-26T02:36:12.011398+0000","last_unstale":"2024-04-26T02:39:31.212798+0000","last_undegraded":"2024-04-26T02:39:31.212798+0000","last_fullsized":"2024-04-26T02:39:31.212798+0000","mapping_epoch":43,"log_start":"28'1600","ondisk_log_start":"28'1600","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11630","last_scrub_stamp":"2024-04-26T02:39:31.212690+0000","last_deep_scrub":"40'11630","last_deep_scrub_stamp":"2024-04-26T02:39:31.212690+0000","last_clean_scrub_stamp":"2024-04-26T02:39:31.212690+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:02:11.585448+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"40'11478","reported_seq":15576,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:31.481834+0000","last_change":"2024-04-26T02:39:31.481865+0000","last_active":"2024-04-26T02:39:31.481834+0000","last_peered":"2024-04-26T02:39:31.481834+0000","last_clean":"2024-04-26T02:39:31.481834+0000","last_became_active":"2024-04-26T02:30:41.914862+0000","last_became_peered":"2024-04-26T02:30:41.914862+0000","last_unstale":"2024-04-26T02:39:31.481834+0000","last_undegraded":"2024-04-26T02:39:31.481834+0000","last_fullsized":"2024-04-26T02:39:31.481834+0000","mapping_epoch":37,"log_start":"28'1400","ondisk_log_start":"28'1400","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11478","last_scrub_stamp":"2024-04-26T02:39:31.481730+0000","last_deep_scrub":"40'11478","last_deep_scrub_stamp":"2024-04-26T02:39:31.481730+0000","last_clean_scrub_stamp":"2024-04-26T02:39:31.481730+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:52:44.271833+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"40'11595","reported_seq":15769,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:33.433201+0000","last_change":"2024-04-26T02:39:33.433253+0000","last_active":"2024-04-26T02:39:33.433201+0000","last_peered":"2024-04-26T02:39:33.433201+0000","last_clean":"2024-04-26T02:39:33.433201+0000","last_became_active":"2024-04-26T02:37:47.074368+0000","last_became_peered":"2024-04-26T02:37:47.074368+0000","last_unstale":"2024-04-26T02:39:33.433201+0000","last_undegraded":"2024-04-26T02:39:33.433201+0000","last_fullsized":"2024-04-26T02:39:33.433201+0000","mapping_epoch":53,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11595","last_scrub_stamp":"2024-04-26T02:39:33.433081+0000","last_deep_scrub":"40'11595","last_deep_scrub_stamp":"2024-04-26T02:39:33.433081+0000","last_clean_scrub_stamp":"2024-04-26T02:39:33.433081+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:23:23.849979+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"40'11600","reported_seq":15829,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:35.476083+0000","last_change":"2024-04-26T02:39:35.476113+0000","last_active":"2024-04-26T02:39:35.476083+0000","last_peered":"2024-04-26T02:39:35.476083+0000","last_clean":"2024-04-26T02:39:35.476083+0000","last_became_active":"2024-04-26T02:39:01.217321+0000","last_became_peered":"2024-04-26T02:39:01.217321+0000","last_unstale":"2024-04-26T02:39:35.476083+0000","last_undegraded":"2024-04-26T02:39:35.476083+0000","last_fullsized":"2024-04-26T02:39:35.476083+0000","mapping_epoch":61,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11600","last_scrub_stamp":"2024-04-26T02:39:35.475977+0000","last_deep_scrub":"40'11600","last_deep_scrub_stamp":"2024-04-26T02:39:35.475977+0000","last_clean_scrub_stamp":"2024-04-26T02:39:35.475977+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:07.601085+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"40'11756","reported_seq":16081,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:30.632592+0000","last_change":"2024-04-26T02:39:30.632618+0000","last_active":"2024-04-26T02:39:30.632592+0000","last_peered":"2024-04-26T02:39:30.632592+0000","last_clean":"2024-04-26T02:39:30.632592+0000","last_became_active":"2024-04-26T02:26:45.793045+0000","last_became_peered":"2024-04-26T02:26:45.793045+0000","last_unstale":"2024-04-26T02:39:30.632592+0000","last_undegraded":"2024-04-26T02:39:30.632592+0000","last_fullsized":"2024-04-26T02:39:30.632592+0000","mapping_epoch":33,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11756","last_scrub_stamp":"2024-04-26T02:39:30.632506+0000","last_deep_scrub":"40'11756","last_deep_scrub_stamp":"2024-04-26T02:39:30.632506+0000","last_clean_scrub_stamp":"2024-04-26T02:39:30.632506+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:56.619263+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'11987","reported_seq":12315,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:33.416589+0000","last_change":"2024-04-26T02:39:33.416624+0000","last_active":"2024-04-26T02:39:33.416589+0000","last_peered":"2024-04-26T02:39:33.416589+0000","last_clean":"2024-04-26T02:39:33.416589+0000","last_became_active":"2024-04-26T02:36:32.127139+0000","last_became_peered":"2024-04-26T02:36:32.127139+0000","last_unstale":"2024-04-26T02:39:33.416589+0000","last_undegraded":"2024-04-26T02:39:33.416589+0000","last_fullsized":"2024-04-26T02:39:33.416589+0000","mapping_epoch":45,"log_start":"28'1900","ondisk_log_start":"28'1900","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11987","last_scrub_stamp":"2024-04-26T02:39:33.416449+0000","last_deep_scrub":"40'11987","last_deep_scrub_stamp":"2024-04-26T02:39:33.416449+0000","last_clean_scrub_stamp":"2024-04-26T02:39:33.416449+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:19:54.127926+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"40'11736","reported_seq":15899,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:31.373604+0000","last_change":"2024-04-26T02:39:31.373763+0000","last_active":"2024-04-26T02:39:31.373604+0000","last_peered":"2024-04-26T02:39:31.373604+0000","last_clean":"2024-04-26T02:39:31.373604+0000","last_became_active":"2024-04-26T02:33:24.359302+0000","last_became_peered":"2024-04-26T02:33:24.359302+0000","last_unstale":"2024-04-26T02:39:31.373604+0000","last_undegraded":"2024-04-26T02:39:31.373604+0000","last_fullsized":"2024-04-26T02:39:31.373604+0000","mapping_epoch":39,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11736","last_scrub_stamp":"2024-04-26T02:39:31.373399+0000","last_deep_scrub":"40'11736","last_deep_scrub_stamp":"2024-04-26T02:39:31.373399+0000","last_clean_scrub_stamp":"2024-04-26T02:39:31.373399+0000","objects_scrubbed":1,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:50:27.959642+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'11562","reported_seq":15811,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:32.267023+0000","last_change":"2024-04-26T02:39:32.267057+0000","last_active":"2024-04-26T02:39:32.267023+0000","last_peered":"2024-04-26T02:39:32.267023+0000","last_clean":"2024-04-26T02:39:32.267023+0000","last_became_active":"2024-04-26T02:38:04.147010+0000","last_became_peered":"2024-04-26T02:38:04.147010+0000","last_unstale":"2024-04-26T02:39:32.267023+0000","last_undegraded":"2024-04-26T02:39:32.267023+0000","last_fullsized":"2024-04-26T02:39:32.267023+0000","mapping_epoch":55,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11562","last_scrub_stamp":"2024-04-26T02:39:32.266905+0000","last_deep_scrub":"40'11562","last_deep_scrub_stamp":"2024-04-26T02:39:32.266905+0000","last_clean_scrub_stamp":"2024-04-26T02:39:32.266905+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:34:16.281525+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"40'11716","reported_seq":15906,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:34.421160+0000","last_change":"2024-04-26T02:39:34.421191+0000","last_active":"2024-04-26T02:39:34.421160+0000","last_peered":"2024-04-26T02:39:34.421160+0000","last_clean":"2024-04-26T02:39:34.421160+0000","last_became_active":"2024-04-26T02:38:42.937589+0000","last_became_peered":"2024-04-26T02:38:42.937589+0000","last_unstale":"2024-04-26T02:39:34.421160+0000","last_undegraded":"2024-04-26T02:39:34.421160+0000","last_fullsized":"2024-04-26T02:39:34.421160+0000","mapping_epoch":59,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11716","last_scrub_stamp":"2024-04-26T02:39:34.421036+0000","last_deep_scrub":"40'11716","last_deep_scrub_stamp":"2024-04-26T02:39:34.421036+0000","last_clean_scrub_stamp":"2024-04-26T02:39:34.421036+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:41.867577+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"40'11596","reported_seq":11874,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:36.424950+0000","last_change":"2024-04-26T02:39:36.424996+0000","last_active":"2024-04-26T02:39:36.424950+0000","last_peered":"2024-04-26T02:39:36.424950+0000","last_clean":"2024-04-26T02:39:36.424950+0000","last_became_active":"2024-04-26T02:38:23.247784+0000","last_became_peered":"2024-04-26T02:38:23.247784+0000","last_unstale":"2024-04-26T02:39:36.424950+0000","last_undegraded":"2024-04-26T02:39:36.424950+0000","last_fullsized":"2024-04-26T02:39:36.424950+0000","mapping_epoch":57,"log_start":"28'1500","ondisk_log_start":"28'1500","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11596","last_scrub_stamp":"2024-04-26T02:39:36.424803+0000","last_deep_scrub":"40'11596","last_deep_scrub_stamp":"2024-04-26T02:39:36.424803+0000","last_clean_scrub_stamp":"2024-04-26T02:39:36.424803+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:35.800116+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"40'11778","reported_seq":12057,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:35.443964+0000","last_change":"2024-04-26T02:39:35.443964+0000","last_active":"2024-04-26T02:39:35.443964+0000","last_peered":"2024-04-26T02:39:35.443964+0000","last_clean":"2024-04-26T02:39:35.443964+0000","last_became_active":"2024-04-26T02:37:28.955354+0000","last_became_peered":"2024-04-26T02:37:28.955354+0000","last_unstale":"2024-04-26T02:39:35.443964+0000","last_undegraded":"2024-04-26T02:39:35.443964+0000","last_fullsized":"2024-04-26T02:39:35.443964+0000","mapping_epoch":51,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11778","last_scrub_stamp":"2024-04-26T02:39:35.443678+0000","last_deep_scrub":"40'11778","last_deep_scrub_stamp":"2024-04-26T02:39:35.443678+0000","last_clean_scrub_stamp":"2024-04-26T02:39:35.443678+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:33.985427+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'12120","reported_seq":12404,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:37.406308+0000","last_change":"2024-04-26T02:39:37.406338+0000","last_active":"2024-04-26T02:39:37.406308+0000","last_peered":"2024-04-26T02:39:37.406308+0000","last_clean":"2024-04-26T02:39:37.406308+0000","last_became_active":"2024-04-26T02:39:19.405863+0000","last_became_peered":"2024-04-26T02:39:19.405863+0000","last_unstale":"2024-04-26T02:39:37.406308+0000","last_undegraded":"2024-04-26T02:39:37.406308+0000","last_fullsized":"2024-04-26T02:39:37.406308+0000","mapping_epoch":63,"log_start":"32'2100","ondisk_log_start":"32'2100","created":13,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12120","last_scrub_stamp":"2024-04-26T02:39:37.406208+0000","last_deep_scrub":"40'12120","last_deep_scrub_stamp":"2024-04-26T02:39:37.406208+0000","last_clean_scrub_stamp":"2024-04-26T02:39:37.406208+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:56.078550+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'11853","reported_seq":16136,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:30.463708+0000","last_change":"2024-04-26T02:39:30.463708+0000","last_active":"2024-04-26T02:39:30.463708+0000","last_peered":"2024-04-26T02:39:30.463708+0000","last_clean":"2024-04-26T02:39:30.463708+0000","last_became_active":"2024-04-26T02:28:52.811840+0000","last_became_peered":"2024-04-26T02:28:52.811840+0000","last_unstale":"2024-04-26T02:39:30.463708+0000","last_undegraded":"2024-04-26T02:39:30.463708+0000","last_fullsized":"2024-04-26T02:39:30.463708+0000","mapping_epoch":35,"log_start":"28'1800","ondisk_log_start":"28'1800","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11853","last_scrub_stamp":"2024-04-26T02:39:30.463590+0000","last_deep_scrub":"40'11853","last_deep_scrub_stamp":"2024-04-26T02:39:30.463590+0000","last_clean_scrub_stamp":"2024-04-26T02:39:30.463590+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:40.427574+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"40'11795","reported_seq":12090,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T02:39:34.394554+0000","last_change":"2024-04-26T02:39:34.394607+0000","last_active":"2024-04-26T02:39:34.394554+0000","last_peered":"2024-04-26T02:39:34.394554+0000","last_clean":"2024-04-26T02:39:34.394554+0000","last_became_active":"2024-04-26T02:37:06.890627+0000","last_became_peered":"2024-04-26T02:37:06.890627+0000","last_unstale":"2024-04-26T02:39:34.394554+0000","last_undegraded":"2024-04-26T02:39:34.394554+0000","last_fullsized":"2024-04-26T02:39:34.394554+0000","mapping_epoch":49,"log_start":"28'1700","ondisk_log_start":"28'1700","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11795","last_scrub_stamp":"2024-04-26T02:39:34.394389+0000","last_deep_scrub":"40'11795","last_deep_scrub_stamp":"2024-04-26T02:39:34.394389+0000","last_clean_scrub_stamp":"2024-04-26T02:39:34.394389+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:47.874547+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6406,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23124,"num_write_kb":92496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4068,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":27,"seq":115964117183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686100,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93685740,"statfs":{"total":96636764160,"available":95934197760,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689748,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93682092,"statfs":{"total":96636764160,"available":95930462208,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":30,"seq":128849019059,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587604,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784236,"statfs":{"total":96636764160,"available":96035057664,"internally_reserved":0,"allocated":667648,"data_stored":336261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215300,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":693819,"kb_avail":93677484,"statfs":{"total":96636764160,"available":95925743616,"internally_reserved":0,"allocated":536576,"data_stored":205189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":710471375},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:39:51.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T02:39:51.970 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T02:39:51.970 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T02:39:51.970 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T02:39:51.970 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:39:52.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-26T02:39:58.072 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T02:39:58.072 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T02:39:58.072 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:39:58.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~6s 2024-04-26T02:40:04.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-26T02:40:04.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-26T02:40:04.174 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T02:40:04.174 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T02:40:04.174 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:09.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-26T02:40:09.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-26T02:40:10.276 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T02:40:10.277 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T02:40:10.277 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:15.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-26T02:40:15.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-26T02:40:16.378 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T02:40:16.378 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T02:40:16.379 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T02:40:16.379 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:16.409 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T02:40:16.409 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T02:40:16.409 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T02:40:16.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:16.467 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T02:40:16.467 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T02:40:16.467 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:16.531 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T02:40:16.532 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T02:40:16.532 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:40:16.592 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T02:40:16.593 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T02:40:16.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-04-26T02:40:16.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:40:16.759 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T02:40:16.975 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:40:16.975 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T02:40:17.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:40:17.222 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T02:40:17.416 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:40:17.416 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T02:40:17.617 INFO:tasks.ceph:Archiving mon data... 2024-04-26T02:40:17.619 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/data/mon.a.tgz 2024-04-26T02:40:17.619 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-26T02:40:17.627 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.kPymb2mcNx 2024-04-26T02:40:17.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kPymb2mcNx 2024-04-26T02:40:17.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.kPymb2mcNx 2024-04-26T02:40:17.869 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.kPymb2mcNx is 308KB 2024-04-26T02:40:17.928 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.kPymb2mcNx 2024-04-26T02:40:17.934 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/data/mon.b.tgz 2024-04-26T02:40:17.934 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-26T02:40:17.982 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.yCc87CTI35 2024-04-26T02:40:17.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yCc87CTI35 2024-04-26T02:40:18.102 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.yCc87CTI35 2024-04-26T02:40:18.167 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.yCc87CTI35 is 312KB 2024-04-26T02:40:18.225 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.yCc87CTI35 2024-04-26T02:40:18.231 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/data/mon.c.tgz 2024-04-26T02:40:18.231 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-26T02:40:18.278 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.DXyux5gzRj 2024-04-26T02:40:18.278 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.DXyux5gzRj 2024-04-26T02:40:18.398 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.DXyux5gzRj 2024-04-26T02:40:18.463 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.DXyux5gzRj is 312KB 2024-04-26T02:40:18.524 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.DXyux5gzRj 2024-04-26T02:40:18.530 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T02:40:18.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T02:40:18.656 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T02:40:18.656 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T02:40:18.658 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/remote/smithi177/crash 2024-04-26T02:40:18.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T02:40:18.711 INFO:tasks.ceph:Compressing logs... 2024-04-26T02:40:18.711 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:40:18.777 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43958.log 2024-04-26T02:40:18.779 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40358.log 2024-04-26T02:40:18.779 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37650.log 2024-04-26T02:40:18.779 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43958.log.gz 2024-04-26T02:40:18.779 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34256.log 2024-04-26T02:40:18.779 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40358.log.gz 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33288.log 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37650.log: /var/log/ceph/ceph-client.admin.34256.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37650.log.gz -- 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.45128.log 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34256.log.gz 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33288.log.gz 2024-04-26T02:40:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47104.log 2024-04-26T02:40:18.785 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45128.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47104.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40123.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37130.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40123.log.gz 2024-04-26T02:40:18.786 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37130.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30319.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36922.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42705.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30319.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31035.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36922.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29923.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42705.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30881.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31035.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29923.log.gz 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45752.log 2024-04-26T02:40:18.787 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30881.log.gz 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29547.log 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47260.log 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45752.log.gz 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29547.log.gz 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43672.log 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42263.log 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47260.log.gz 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34014.log 2024-04-26T02:40:18.798 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43672.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30143.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25204.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34014.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34254.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30143.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31148.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25204.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33009.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34254.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47547.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31148.log.gz 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39055.log 2024-04-26T02:40:18.799 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33009.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34404.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47547.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34619.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39055.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34404.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34619.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40800.log: /var/log/ceph/ceph-client.admin.37390.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36296.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30903.log 2024-04-26T02:40:18.800 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37390.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30793.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36296.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34334.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30903.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30793.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38664.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34334.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46272.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40071.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38664.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46272.log.gz 2024-04-26T02:40:18.801 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45336.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40071.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47416.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33832.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45336.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37442.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47416.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33832.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37442.log.gz 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-04-26T02:40:18.802 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36948.log 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36948.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36948.log.gz 2024-04-26T02:40:18.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34330.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35103.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34330.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45284.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37728.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35103.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45284.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25252.log 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37728.log.gz 2024-04-26T02:40:18.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38508.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25252.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46662.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46090.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38508.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46662.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38014.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46090.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44114.log 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2024-04-26T02:40:18.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38612.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38014.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46922.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44114.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38612.log: -- /var/log/ceph/ceph-client.admin.42968.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38612.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46922.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46922.log.gz -5 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47912.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34040.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42968.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46402.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47912.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44322.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34040.log.gz 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-04-26T02:40:18.832 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46402.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5/var/log/ceph/ceph-client.admin.44322.log: --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44322.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32929.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29726.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32929.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37338.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42575.log 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29726.log.gz 2024-04-26T02:40:18.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37338.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29597.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47729.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29804.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29597.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34432.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47729.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29804.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30187.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34432.log.gz 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31123.log 2024-04-26T02:40:18.834 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30187.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.31123.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39523.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31123.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43776.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47625.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39523.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29244.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43776.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45258.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47625.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46480.log 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29244.log.gz 2024-04-26T02:40:18.835 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.45258.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45258.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38222.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46480.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38794.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38222.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29233.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38794.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44686.log 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2024-04-26T02:40:18.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34685.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29233.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33002.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44686.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34685.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33884.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33002.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30859.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31057.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33884.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30859.log.gz 2024-04-26T02:40:18.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47677.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31057.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42601.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34905.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47677.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42601.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44530.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34905.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43802.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5/var/log/ceph/ceph-client.admin.44530.log: --verbose -- /var/log/ceph/ceph-client.admin.47808.log 2024-04-26T02:40:18.838 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44530.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38534.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43802.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45648.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47808.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45778.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38534.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46610.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45648.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33936.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45778.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.46610.log: /var/log/ceph/ceph-client.admin.37832.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46610.log.gz 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.33936.log: -- /var/log/ceph/ceph-client.admin.46168.log 2024-04-26T02:40:18.839 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33936.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30165.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37832.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29879.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46168.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30165.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45024.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29879.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42159.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.45024.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47964.log 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45024.log.gz 2024-04-26T02:40:18.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37754.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42159.log.gz 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47964.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47964.log.gz --verbose 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.44270.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37754.log.gz 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46740.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44270.log.gz 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gzgzip 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33988.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.46740.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46740.log.gz 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33988.log.gz 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29129.log 2024-04-26T02:40:18.841 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2024-04-26T02:40:19.019 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-04-26T02:40:19.019 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44972.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29129.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.44972.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44972.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46792.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43204.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46792.log.gz 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34597.log 2024-04-26T02:40:19.020 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43204.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36166.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34597.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39315.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz -- 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.30749.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36166.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30749.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38560.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29245.log 2024-04-26T02:40:19.021 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41715.log/var/log/ceph/ceph-client.admin.38560.log: 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38560.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29245.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47000.log 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41715.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37884.log 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47000.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38248.log 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37884.log.gz 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42627.log 2024-04-26T02:40:19.022 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38248.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42627.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.42627.log.gz /var/log/ceph/ceph-client.admin.33494.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39861.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33494.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34949.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43464.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39861.log: /var/log/ceph/ceph-client.admin.34949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39861.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34949.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36896.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48614.log 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43464.log.gz 2024-04-26T02:40:19.023 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47755.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36896.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48614.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47755.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38326.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41923.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38326.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39757.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41923.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45206.log 2024-04-26T02:40:19.024 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38586.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39757.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45206.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48146.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38586.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43282.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48146.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44062.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44426.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43282.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37572.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44062.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42994.log 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44426.log.gz 2024-04-26T02:40:19.025 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37988.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37572.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42679.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42994.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38951.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37988.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47573.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42679.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34531.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38951.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38482.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47573.log.gz 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36400.log 2024-04-26T02:40:19.026 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34531.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29034.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38482.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34222.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36400.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29034.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29901.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34222.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42079.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29901.log.gz 2024-04-26T02:40:19.027 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33140.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42079.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30011.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45180.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33140.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39575.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30011.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45180.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39575.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45388.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32872.log 2024-04-26T02:40:19.028 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45388.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32872.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42185.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36033.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42185.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42185.log.gz -5 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29363.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.36033.log: /var/log/ceph/ceph-client.admin.44582.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36033.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30363.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29363.log.gz 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34553.log 2024-04-26T02:40:19.029 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44582.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38742.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30363.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34553.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30815.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38742.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35037.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30815.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29141.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35037.log.gz 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37858.log 2024-04-26T02:40:19.030 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34118.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29141.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33468.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37858.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36766.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34118.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45726.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33468.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36766.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45726.log.gz 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36374.log 2024-04-26T02:40:19.031 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37936.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42890.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36374.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37468.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37936.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36688.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42890.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log/var/log/ceph/ceph-client.admin.37468.log: 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37468.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37962.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36688.log.gz 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-04-26T02:40:19.032 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37962.log: -- /var/log/ceph/ceph-client.admin.42916.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37962.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41559.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45882.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42916.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38846.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41559.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47182.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45882.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36532.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38846.log.gz 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47182.log: /var/log/ceph/ceph-client.admin.31079.log 2024-04-26T02:40:19.033 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47182.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46220.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36532.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31079.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31079.log.gz -5 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46064.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46220.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36085.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48744.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46064.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25314.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36085.log.gz 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2024-04-26T02:40:19.034 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48744.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33080.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25314.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43646.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36818.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33080.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43646.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45934.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36818.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33650.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31101.log 2024-04-26T02:40:19.035 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45934.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44400.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33650.log: /var/log/ceph/ceph-client.admin.31101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33650.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31101.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46558.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44400.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44400.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.30991.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37182.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46558.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30991.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29674.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37182.log.gz 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42942.log 2024-04-26T02:40:19.036 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42757.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29674.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38378.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42942.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38378.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31013.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30727.log 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31013.log.gz 2024-04-26T02:40:19.037 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39835.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.30727.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30727.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39835.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37364.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37286.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37364.log.gz 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37286.log.gzgzip 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45232.log 2024-04-26T02:40:19.038 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33858.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42105.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45232.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33858.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48068.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44946.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42105.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43568.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48068.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44946.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38196.log 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43568.log.gz 2024-04-26T02:40:19.039 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38274.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33598.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38196.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41949.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38274.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33598.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43906.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41949.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39679.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36714.log 2024-04-26T02:40:19.040 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43906.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39679.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30033.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36714.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42393.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30033.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34795.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42393.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32932.log 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34795.log.gz 2024-04-26T02:40:19.041 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34663.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25276.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32932.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34663.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34663.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45414.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25276.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47495.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36270.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45414.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34861.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47495.log.gz 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48328.log 2024-04-26T02:40:19.042 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36270.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41767.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34861.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43516.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48328.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40774.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41767.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47130.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43516.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44192.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40774.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47130.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47130.log.gz -5 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34092.log 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44192.log.gz 2024-04-26T02:40:19.043 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43698.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34092.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.43698.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43698.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37208.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39965.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37208.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39497.log 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-04-26T02:40:19.044 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39965.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39497.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40514.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47156.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40514.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.47156.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43072.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47156.log.gz 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34641.log 2024-04-26T02:40:19.045 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29441.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43072.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34817.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34641.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33442.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29441.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48276.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34817.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48796.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33442.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48276.log.gz 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30639.log 2024-04-26T02:40:19.046 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48796.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46324.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47782.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30639.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34707.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46324.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44036.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47782.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36870.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44036.log.gz 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-04-26T02:40:19.047 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39419.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36870.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47338.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38430.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39419.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47338.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47860.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38430.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29237.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48846.log 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47860.log.gz 2024-04-26T02:40:19.048 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44140.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29237.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48846.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30617.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44140.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30617.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30617.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29830.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45804.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37052.log 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29830.log.gz 2024-04-26T02:40:19.049 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34170.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36140.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37052.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.34170.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45830.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34170.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36140.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36322.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45830.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47938.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36322.log.gz 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-04-26T02:40:19.050 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30077.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47938.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48458.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30077.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48458.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20244.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48458.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45908.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.tmp-client.admin.20244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37702.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20244.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45908.log: -- /var/log/ceph/ceph-client.admin.47208.log 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45908.log.gz 2024-04-26T02:40:19.051 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37702.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37702.log.gz -- 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.33210.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30385.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47208.log.gz 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.33210.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33210.log.gz 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30385.log.gz 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37910.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30771.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz --verbose 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.29989.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37910.log.gz 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2024-04-26T02:40:19.052 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29989.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43490.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33962.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43490.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40043.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33962.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-04-26T02:40:19.053 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40043.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42835.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25147.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42835.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42835.log.gz 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33676.log 2024-04-26T02:40:19.054 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25147.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47052.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33676.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48354.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47052.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45986.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30253.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48354.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34839.log 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45986.log.gz 2024-04-26T02:40:19.055 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39341.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30253.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34839.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48172.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39341.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45102.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48172.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29493.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45102.log.gz 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-04-26T02:40:19.056 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46142.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29493.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46506.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33296.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46142.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43438.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46506.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33296.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43438.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2024-04-26T02:40:19.057 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33390.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48198.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42237.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44738.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48198.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40097.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42237.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44738.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40097.log.gz 2024-04-26T02:40:19.058 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46532.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34927.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46532.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46532.log.gz -5 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34927.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33572.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47390.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33572.log.gz 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-04-26T02:40:19.059 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-mgr.x.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46948.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36610.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46948.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48380.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36610.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48380.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34482.log 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2024-04-26T02:40:19.060 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45466.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34482.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47834.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45466.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45466.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30705.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47834.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45674.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33702.log 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30705.log.gz 2024-04-26T02:40:19.061 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48120.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45674.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33702.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48120.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33806.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43880.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42523.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33806.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39107.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43880.log.gz 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37624.log 2024-04-26T02:40:19.062 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42523.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39107.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37624.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37546.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48250.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46974.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37546.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48250.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46974.log.gz 2024-04-26T02:40:19.063 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43828.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46246.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25003.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43828.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36480.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46246.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25003.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37676.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36480.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42315.log 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37676.log: 95.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T02:40:19.064 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.37676.log.gz -- 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.40017.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36244.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42315.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47286.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44764.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36244.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37520.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47286.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38690.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44764.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37520.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37520.log.gz 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47312.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47078.log 2024-04-26T02:40:19.065 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38690.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33624.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47312.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29730.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47078.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44348.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33624.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43098.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29730.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46688.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44348.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43098.log.gz 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45544.log 2024-04-26T02:40:19.066 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46688.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30476.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44816.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45544.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30476.log: -- /var/log/ceph/ceph-client.admin.42001.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30476.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43020.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44816.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42001.log: -- /var/log/ceph/ceph-client.admin.45700.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42001.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37494.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43020.log.gz 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45440.log 2024-04-26T02:40:19.067 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29140.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37494.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45440.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30275.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29140.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34408.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39263.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30275.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29625.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34408.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29036.log 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39263.log.gz 2024-04-26T02:40:19.068 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36218.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29625.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29036.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30969.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36218.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33218.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48692.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30969.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48692.log.gz 2024-04-26T02:40:19.069 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42783.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44634.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42211.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42783.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39913.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44634.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42211.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33366.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39913.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43854.log 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2024-04-26T02:40:19.070 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43046.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33366.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43854.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42132.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43046.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39809.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42132.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47234.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39809.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33910.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34066.log 2024-04-26T02:40:19.071 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47234.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37078.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33910.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43308.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34066.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37078.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34196.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43308.log: /var/log/ceph/ceph-client.admin.41689.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43308.log.gz -5 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46012.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29626.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34196.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29678.log 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46012.log.gz 2024-04-26T02:40:19.072 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29626.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40592.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37156.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36114.log 0.0% 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40592.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48718.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37156.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36114.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47442.log 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48718.log.gz 2024-04-26T02:40:19.073 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38144.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44998.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47442.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38144.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35955.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44998.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29415.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35955.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35955.log.gz 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45518.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29415.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T02:40:19.074 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29415.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45518.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48588.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30341.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39939.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48588.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30341.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39939.log.gz 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-04-26T02:40:19.075 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43230.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29519.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44842.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43230.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39133.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29519.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30297.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44842.log.gz 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-04-26T02:40:19.076 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39133.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30297.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30451.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25397.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30451.log.gz 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36007.log 2024-04-26T02:40:19.077 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25397.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37000.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36007.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30429.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39653.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37000.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30429.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48640.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39653.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33106.log 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-04-26T02:40:19.078 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48640.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 96.4% -- replaced with /var/log/ceph/ceph-client.admin.33390.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30121.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33106.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44374.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32931.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30121.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29031.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44374.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31174.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32931.log.gz 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42731.log 2024-04-26T02:40:19.079 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29031.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31174.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44088.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42731.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33416.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44088.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44790.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33416.log.gz 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33520.log 2024-04-26T02:40:19.080 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44790.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34575.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33520.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34575.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47026.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44868.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38300.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47026.log.gz 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46428.log 2024-04-26T02:40:19.081 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44868.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36636.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38300.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33780.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46428.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30595.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36636.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33780.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29967.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30595.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43412.log 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2024-04-26T02:40:19.082 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42027.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29967.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32898.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43412.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37234.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42027.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44504.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32898.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42471.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37234.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44504.log.gz 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37026.log 2024-04-26T02:40:19.083 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42471.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30661.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47599.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37026.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29945.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30661.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47599.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29389.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30528.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29945.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29389.log.gz 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39887.log 2024-04-26T02:40:19.084 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30528.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30099.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39887.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36974.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30099.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45362.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37598.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36974.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46714.log 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45362.log.gz 2024-04-26T02:40:19.085 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46376.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37598.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30837.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29467.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46376.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30837.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29467.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44920.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47521.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44920.log.gz 2024-04-26T02:40:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47521.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47886.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30209.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 95.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47886.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44712.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30209.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40202.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2024-04-26T02:40:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34324.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44712.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47364.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33546.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34324.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47364.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29337.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33546.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47703.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29337.log.gz 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46844.log 2024-04-26T02:40:19.088 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46194.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47703.log.gz 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38456.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46844.log.gz 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46194.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46194.log.gz -5 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38456.log.gz 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28947.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44660.log 2024-04-26T02:40:19.089 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28947.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48484.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44660.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41845.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46818.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48484.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.41845.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28920.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41845.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37416.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46818.log.gz 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30947.log 2024-04-26T02:40:19.090 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28920.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37416.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30947.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30573.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30947.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43724.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33754.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43724.log.gz 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz --verbose 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.40436.log 2024-04-26T02:40:19.091 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43984.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33754.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43334.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43984.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30407.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43334.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43750.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44452.log 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30407.log.gz 2024-04-26T02:40:19.092 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38352.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43750.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44452.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29574.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38352.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33010.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29574.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29778.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33010.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30925.log 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-04-26T02:40:19.093 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29778.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30683.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36426.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30925.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46350.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30683.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36426.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46350.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44218.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33295.log 2024-04-26T02:40:19.094 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44218.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29856.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33295.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35081.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29856.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25228.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35081.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37104.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.25228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28999.log 2024-04-26T02:40:19.095 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25228.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43176.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37104.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28999.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37260.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43176.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.40906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46870.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43620.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37260.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.46870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46870.log.gz 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43620.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.43620.log.gz --verbose 2024-04-26T02:40:19.096 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.30502.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34993.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36844.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30502.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36662.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34993.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36792.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36844.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48510.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36662.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36792.log.gz 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43932.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36452.log 2024-04-26T02:40:19.097 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48510.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38404.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43932.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34144.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36452.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38768.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38404.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34144.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.38768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43124.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38768.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37312.log 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-04-26T02:40:19.098 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.43124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43124.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45856.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37312.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48302.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.45856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36558.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45856.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48536.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48302.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48406.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36558.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37806.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48536.log.gz 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.48406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36348.log 2024-04-26T02:40:19.099 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48406.log.gz 2024-04-26T02:40:19.100 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.37806.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37806.log.gz /var/log/ceph/ceph-client.admin.44244.log 2024-04-26T02:40:19.100 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-26T02:40:19.100 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.36348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36348.log.gz 2024-04-26T02:40:19.100 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.44244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44244.log.gz 2024-04-26T02:40:20.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-26T02:40:20.703 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T02:40:21.236 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T02:40:22.137 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T02:40:22.785 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T02:49:21.655 INFO:teuthology.orchestra.run.smithi177.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T02:53:51.959 INFO:teuthology.orchestra.run.smithi177.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T02:54:08.255 INFO:teuthology.orchestra.run.smithi177.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T02:54:31.910 INFO:teuthology.orchestra.run.smithi177.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T02:54:31.911 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-26T02:54:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:real 14m13.154s 2024-04-26T02:54:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:user 16m8.424s 2024-04-26T02:54:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m43.904s 2024-04-26T02:54:31.912 INFO:tasks.ceph:Archiving logs... 2024-04-26T02:54:31.913 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/remote/smithi177/log 2024-04-26T02:54:31.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:55:24.177 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T02:55:24.194 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T02:55:24.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:55:24.257 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T02:55:24.549 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T02:55:24.549 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout:+hv01.front.sepi 67.205.162.81 3 u 16 64 377 0.109 -7.389 1.084 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 64 377 0.100 -3.043 1.514 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout:*hv03.front.sepi 74.6.168.72 3 u 29 64 377 0.090 -5.309 1.439 2024-04-26T02:55:24.601 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-26T02:55:24.602 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T02:55:24.632 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T02:55:24.633 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T02:55:24.653 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T02:55:24.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T02:55:24.718 INFO:teuthology.task.internal:Duration was 2477.730919 seconds 2024-04-26T02:55:24.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T02:55:24.734 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T02:55:24.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:55:24.935 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T02:55:24.935 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-26T02:55:24.935 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:55:24.946 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T02:55:24.946 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:55:25.030 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T02:55:25.031 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:55:25.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T02:55:25.485 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T02:55:25.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:55:25.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T02:55:25.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:55:25.753 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-26T02:55:25.785 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:55:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:55:25.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T02:55:26.018 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T02:55:26.019 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674290/remote/smithi177 2024-04-26T02:55:26.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:55:26.103 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T02:55:26.103 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:55:26.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T02:55:26.159 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T02:55:26.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T02:55:26.182 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T02:55:26.182 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:55:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:55 /home/ubuntu/cephtest 2024-04-26T02:55:26.196 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T02:55:26.209 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T02:55:26.242 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} duration: 2477.7309188842773 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-26T02:55:26.242 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:55:26.375 INFO:teuthology.run:pass